{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5a1368f0",
   "metadata": {},
   "source": [
    "# 作业4：卷积神经网络实践"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "294a04c2",
   "metadata": {},
   "source": [
    "本次作业将练习卷积神经网络，利用卷积层和全连接层实现手写数字的识别。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9ee659c",
   "metadata": {},
   "source": [
    "## 1. 目标"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "997082ed",
   "metadata": {},
   "source": [
    "通过对 MNIST 数据进行训练，构建一个简单的图像分类模型，对图片中的数字进行识别。你将利用该模型对自己真实手写出的数字进行预测，观察模型效果。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0490f78e",
   "metadata": {},
   "source": [
    "## 2. 主要步骤"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b786836",
   "metadata": {},
   "source": [
    "1. 获取数据\n",
    "2. 定义模型结构\n",
    "3. 创建模型类\n",
    "4. 定义损失函数\n",
    "5. 编写训练循环\n",
    "6. 实施预测"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45cd7d88",
   "metadata": {},
   "source": [
    "### 2.1 获取数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccc63241",
   "metadata": {},
   "source": [
    "我们使用知名的 MNIST 数据集，它可以从 PyTorch 中利用工具函数下载得到。MNIST 数据训练集大小为60000，我们将**使用完整训练集进行训练**，并对10个测试集观测进行预测展示。以下函数会在当前目录建立一个名为 data 的文件夹，其中会包含下载得到的数据集。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2069eef6",
   "metadata": {},
   "source": [
    "**注意：请在任何程序的最开始加上随机数种子的设置。请保持这一习惯。**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "57301cf0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
      "Failed to download (trying next):\n",
      "HTTP Error 403: Forbidden\n",
      "\n",
      "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz\n",
      "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz to data\\MNIST\\raw\\train-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████| 9912422/9912422 [00:08<00:00, 1189088.77it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data\\MNIST\\raw\\train-images-idx3-ubyte.gz to data\\MNIST\\raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
      "Failed to download (trying next):\n",
      "HTTP Error 403: Forbidden\n",
      "\n",
      "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz\n",
      "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz to data\\MNIST\\raw\\train-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████| 28881/28881 [00:00<00:00, 130624.24it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data\\MNIST\\raw\\train-labels-idx1-ubyte.gz to data\\MNIST\\raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
      "Failed to download (trying next):\n",
      "HTTP Error 403: Forbidden\n",
      "\n",
      "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz\n",
      "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz to data\\MNIST\\raw\\t10k-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████| 1648877/1648877 [00:01<00:00, 1057648.95it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data\\MNIST\\raw\\t10k-images-idx3-ubyte.gz to data\\MNIST\\raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
      "Failed to download (trying next):\n",
      "HTTP Error 403: Forbidden\n",
      "\n",
      "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz\n",
      "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz to data\\MNIST\\raw\\t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████| 4542/4542 [00:00<00:00, 2259581.16it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data\\MNIST\\raw\\t10k-labels-idx1-ubyte.gz to data\\MNIST\\raw\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "from torchvision import datasets\n",
    "from torchvision.transforms import ToTensor\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "np.random.seed(123456)\n",
    "torch.manual_seed(123456)\n",
    "\n",
    "mnist = datasets.MNIST(\n",
    "    root=\"data\",\n",
    "    train=True,\n",
    "    download=True,\n",
    "    transform=ToTensor()\n",
    ")\n",
    "loader = DataLoader(mnist, batch_size=60000, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afdaba44",
   "metadata": {},
   "source": [
    "我们一次性取出60000个观测，其中 x 是图片数据，y 是图片对应的数字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "61ae2977",
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = next(iter(loader))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "145971d1",
   "metadata": {},
   "source": [
    "一个习惯性动作是查看数据的大小和维度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d67073f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([60000, 1, 28, 28])\n",
      "torch.Size([60000])\n"
     ]
    }
   ],
   "source": [
    "print(x.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fbb19ee",
   "metadata": {},
   "source": [
    "用类似的方法获取测试集，并取出10个观测："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "25db4998",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([10, 1, 28, 28])\n",
      "torch.Size([10])\n"
     ]
    }
   ],
   "source": [
    "mnist_test = datasets.MNIST(\n",
    "    root=\"data\",\n",
    "    train=False,\n",
    "    download=True,\n",
    "    transform=ToTensor()\n",
    ")\n",
    "loader = DataLoader(mnist_test, batch_size=10, shuffle=True)\n",
    "\n",
    "xtest, ytest = next(iter(loader))\n",
    "print(xtest.shape)\n",
    "print(ytest.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8a0aa24",
   "metadata": {},
   "source": [
    "我们可以利用下面的函数展示图片的内容。如选择第一张测试图片，先将其转换成 Numpy 数组，再绘制图形："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c2de2307",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 28)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZg0lEQVR4nO3dbWxT593H8Z+B1DwseIogsTNClm2wTQ1jKzAg4rEtEdHKSmk72kpVeIPoCExZWnVjaCLdJNKByvYig2rdxEArKy8GFAnUNhMkUIVUKYKV0gqlIiypIErJmB0ChFGu+wXCuk3CwzF2/rbz/UhHwsfnyrlyesqXE9snPuecEwAABoZYTwAAMHgRIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYGaY9QRudf36dZ09e1bZ2dny+XzW0wEAeOScU3d3t/Lz8zVkyJ2vdVIuQmfPnlVBQYH1NAAA96m9vV3jxo274zYp9+O47Oxs6ykAABLgXv4+T1qENm/erKKiIg0fPlxTpkzR4cOH72kcP4IDgMxwL3+fJyVCO3fuVGVlpdauXatjx45p9uzZKisrU1tbWzJ2BwBIU75k3EV7+vTpeuihh7Rly5bouu9+97tavHixampq7jg2EokoEAgkekoAgAEWDoc1evToO26T8Cuhq1ev6ujRoyotLY1ZX1paqsbGxj7b9/b2KhKJxCwAgMEh4RE6f/68vvzyS+Xl5cWsz8vLU0dHR5/ta2pqFAgEogvvjAOAwSNpb0y49QUp51y/L1KtWbNG4XA4urS3tydrSgCAFJPwzwmNGTNGQ4cO7XPV09nZ2efqSJL8fr/8fn+ipwEASAMJvxJ64IEHNGXKFNXV1cWsr6urU0lJSaJ3BwBIY0m5Y0JVVZWef/55TZ06VTNnztSf/vQntbW16YUXXkjG7gAAaSopEVq6dKm6urr0m9/8RufOnVNxcbH279+vwsLCZOwOAJCmkvI5ofvB54QAIDOYfE4IAIB7RYQAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJgZZj0BAMmzaNGiuMbt2bPH85h9+/Z5HvPjH//Y8xhkFq6EAABmiBAAwEzCI1RdXS2fzxezBIPBRO8GAJABkvKa0IMPPqh//vOf0cdDhw5Nxm4AAGkuKREaNmwYVz8AgLtKymtCLS0tys/PV1FRkZ555hmdPn36ttv29vYqEonELACAwSHhEZo+fbq2b9+ud999V2+88YY6OjpUUlKirq6ufrevqalRIBCILgUFBYmeEgAgRSU8QmVlZXryySc1adIkPfroo9HPDmzbtq3f7desWaNwOBxd2tvbEz0lAECKSvqHVUeNGqVJkyappaWl3+f9fr/8fn+ypwEASEFJ/5xQb2+vPv30U4VCoWTvCgCQZhIeoZdeekkNDQ1qbW3VBx98oKeeekqRSETl5eWJ3hUAIM0l/Mdxn3/+uZ599lmdP39eY8eO1YwZM9TU1KTCwsJE7woAkOYSHqG33nor0V8SKWrOnDmex7zyyiuexzz//POex3z++eeex2Sin/zkJ3GNc84NyBiAe8cBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaS/kvtkLlyc3M9j5k9e7bnMW+//bbnMbNmzfI8RpIuX74c17iB8O1vf9vzmJKSkiTMBEgcroQAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghrtoI24bN24ckP1MnjzZ85isrKy49pXKd9EeO3as5zGFhYVJmAmQOFwJAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmuIEpkCZ++9vfWk/hjj7++GPrKSANcSUEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJjhBqbQ008/Hde4MWPGeB5z7do1z2M2b97secylS5c8j8H9ef31162ngDTElRAAwAwRAgCY8RyhQ4cOadGiRcrPz5fP59OePXtinnfOqbq6Wvn5+RoxYoTmzZunkydPJmq+AIAM4jlCPT09mjx5smpra/t9fsOGDdq0aZNqa2vV3NysYDCoBQsWqLu7+74nCwDILJ7fmFBWVqaysrJ+n3PO6Q9/+IPWrl2rJUuWSJK2bdumvLw87dixQytWrLi/2QIAMkpCXxNqbW1VR0eHSktLo+v8fr/mzp2rxsbGfsf09vYqEonELACAwSGhEero6JAk5eXlxazPy8uLPnermpoaBQKB6FJQUJDIKQEAUlhS3h3n8/liHjvn+qy7ac2aNQqHw9Glvb09GVMCAKSghH5YNRgMSrpxRRQKhaLrOzs7+1wd3eT3++X3+xM5DQBAmkjolVBRUZGCwaDq6uqi665evaqGhgaVlJQkclcAgAzg+Uro4sWL+uyzz6KPW1tbdfz4ceXk5Gj8+PGqrKzU+vXrNWHCBE2YMEHr16/XyJEj9dxzzyV04gCA9Oc5Qh9++KHmz58ffVxVVSVJKi8v11//+le9/PLLunz5slauXKkLFy5o+vTpeu+995SdnZ24WQMAMoLPOeesJ/H/RSIRBQIB62mkreHDh3ses2/fvrj2NXfuXM9j2traPI/5xje+4XlMJtq4caPnMT//+c/j2ld9fb3nMTc/G+gFH8nIbOFwWKNHj77jNtw7DgBghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGYS+ptVYa+2ttbzmHjuhh2v5ubmAdtXplmwYMGA7evIkSOexwzUHbG///3vex4zcuTIuPbV2NgY1zjcO66EAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAz3MA0hX3ve9/zPOZHP/pREmaSOHV1dQOyn69+9atxjfvf//7neUxPT09c+4L09a9/3fOYiooKz2Mee+wxz2Ok+G4a+/HHH8e1r8GKKyEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAw3ME1ho0eP9jxm7NixSZhJ4mzcuNHzmB/84Aeex3zrW9/yPEaSTp065XnMz372s7j2lcoWLlzoeUxOTo7nMWVlZZ7HjB8/3vOY5uZmz2Mk6fz583GNw73jSggAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMONzzjnrSfx/kUhEgUDAehopIZ6bcO7bt29A9oP7M2SI93//Xb9+PQkzST9NTU2exzzyyCNx7evKlStxjcMN4XD4rjdi5koIAGCGCAEAzHiO0KFDh7Ro0SLl5+fL5/Npz549Mc8vW7ZMPp8vZpkxY0ai5gsAyCCeI9TT06PJkyertrb2ttssXLhQ586diy779++/r0kCADKT59+sWlZWdtffhuj3+xUMBuOeFABgcEjKa0L19fXKzc3VxIkTtXz5cnV2dt52297eXkUikZgFADA4JDxCZWVlevPNN3XgwAG99tpram5u1sMPP6ze3t5+t6+pqVEgEIguBQUFiZ4SACBFef5x3N0sXbo0+ufi4mJNnTpVhYWF2rdvn5YsWdJn+zVr1qiqqir6OBKJECIAGCQSHqFbhUIhFRYWqqWlpd/n/X6//H5/sqcBAEhBSf+cUFdXl9rb2xUKhZK9KwBAmvF8JXTx4kV99tln0cetra06fvy4cnJylJOTo+rqaj355JMKhUI6c+aMfvWrX2nMmDF64oknEjpxAED68xyhDz/8UPPnz48+vvl6Tnl5ubZs2aITJ05o+/bt+u9//6tQKKT58+dr586dys7OTtysAQAZgRuYZph4bkb65z//OQkzsRXvXTqGDUv6y6SSJJ/P53lMiv2v2sd//vMfz2Pef/99z2NWrFjhecwXX3zheQzuHzcwBQCkNCIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJjhLtrISE899VRc41L5Ltrr16/3PKagoMDzmHg9/fTTnsfs3r07CTNBquAu2gCAlEaEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmOEGpkCaOH78uOcxxcXFiZ/IbRQVFXke097enoSZIFVwA1MAQEojQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzHiKUE1NjaZNm6bs7Gzl5uZq8eLFOnXqVMw2zjlVV1crPz9fI0aM0Lx583Ty5MmEThoAkBk8RaihoUEVFRVqampSXV2drl27ptLSUvX09ES32bBhgzZt2qTa2lo1NzcrGAxqwYIF6u7uTvjkAQDpzeecc/EO/uKLL5Sbm6uGhgbNmTNHzjnl5+ersrJSv/jFLyRJvb29ysvL0+9+9zutWLHirl8zEokoEAjEOyUgYx0/ftzzmOLi4sRP5DaKioo8j2lvb0/CTJAqwuGwRo8efcdt7us1oXA4LEnKycmRJLW2tqqjo0OlpaXRbfx+v+bOnavGxsZ+v0Zvb68ikUjMAgAYHOKOkHNOVVVVmjVrVvRfWx0dHZKkvLy8mG3z8vKiz92qpqZGgUAguhQUFMQ7JQBAmok7QqtWrdJHH32kv//9732e8/l8MY+dc33W3bRmzRqFw+HowuU5AAwew+IZtHr1au3du1eHDh3SuHHjouuDwaCkG1dEoVAour6zs7PP1dFNfr9ffr8/nmkAANKcpysh55xWrVqlXbt26cCBA31eiCwqKlIwGFRdXV103dWrV9XQ0KCSkpLEzBgAkDE8XQlVVFRox44devvtt5WdnR19nScQCGjEiBHy+XyqrKzU+vXrNWHCBE2YMEHr16/XyJEj9dxzzyXlGwAApC9PEdqyZYskad68eTHrt27dqmXLlkmSXn75ZV2+fFkrV67UhQsXNH36dL333nvKzs5OyIQBAJnjvj4nlAx8TgjoH58TQrpJ+ueEAAC4H0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADDDXbSBNPHoo496HvPOO+8kYSb9O3LkiOcxmzdv9jzmX//6l+cxn3zyiecxuH/cRRsAkNKIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADPcwBRIE3e7EWR/fv/738e1r/Ly8rjGDYQPPvjA85hHHnkkrn1duXIlrnG4gRuYAgBSGhECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghhuYAhnM7/fHNW7ZsmWexxQXF3ses3TpUs9jSktLPY85fvy45zG4f9zAFACQ0ogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM9zAFACQFNzAFACQ0ogQAMCMpwjV1NRo2rRpys7OVm5urhYvXqxTp07FbLNs2TL5fL6YZcaMGQmdNAAgM3iKUENDgyoqKtTU1KS6ujpdu3ZNpaWl6unpidlu4cKFOnfuXHTZv39/QicNAMgMw7xs/M4778Q83rp1q3Jzc3X06FHNmTMnut7v9ysYDCZmhgCAjHVfrwmFw2FJUk5OTsz6+vp65ebmauLEiVq+fLk6Oztv+zV6e3sViURiFgDA4BD3W7Sdc3r88cd14cIFHT58OLp+586d+spXvqLCwkK1trbq17/+ta5du6ajR4/2+/vuq6ur9corr8T/HQAAUtK9vEVbLk4rV650hYWFrr29/Y7bnT171mVlZbl//OMf/T5/5coVFw6Ho0t7e7uTxMLCwsKS5ks4HL5rSzy9JnTT6tWrtXfvXh06dEjjxo2747ahUEiFhYVqaWnp93m/39/vFRIAIPN5ipBzTqtXr9bu3btVX1+voqKiu47p6upSe3u7QqFQ3JMEAGQmT29MqKio0N/+9jft2LFD2dnZ6ujoUEdHhy5fvixJunjxol566SUdOXJEZ86cUX19vRYtWqQxY8boiSeeSMo3AABIY15eB9Jtfu63detW55xzly5dcqWlpW7s2LEuKyvLjR8/3pWXl7u2trZ73kc4HDb/OSYLCwsLy/0v9/KaEDcwBQAkBTcwBQCkNCIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAmZSLkHPOegoAgAS4l7/PUy5C3d3d1lMAACTAvfx97nMpdulx/fp1nT17VtnZ2fL5fDHPRSIRFRQUqL29XaNHjzaaoT2Oww0chxs4DjdwHG5IhePgnFN3d7fy8/M1ZMidr3WGDdCc7tmQIUM0bty4O24zevToQX2S3cRxuIHjcAPH4QaOww3WxyEQCNzTdin34zgAwOBBhAAAZtIqQn6/X+vWrZPf77eeiimOww0chxs4DjdwHG5It+OQcm9MAAAMHml1JQQAyCxECABghggBAMwQIQCAmbSK0ObNm1VUVKThw4drypQpOnz4sPWUBlR1dbV8Pl/MEgwGraeVdIcOHdKiRYuUn58vn8+nPXv2xDzvnFN1dbXy8/M1YsQIzZs3TydPnrSZbBLd7TgsW7asz/kxY8YMm8kmSU1NjaZNm6bs7Gzl5uZq8eLFOnXqVMw2g+F8uJfjkC7nQ9pEaOfOnaqsrNTatWt17NgxzZ49W2VlZWpra7Oe2oB68MEHde7cuehy4sQJ6yklXU9PjyZPnqza2tp+n9+wYYM2bdqk2tpaNTc3KxgMasGCBRl3H8K7HQdJWrhwYcz5sX///gGcYfI1NDSooqJCTU1Nqqur07Vr11RaWqqenp7oNoPhfLiX4yClyfng0sQPf/hD98ILL8Ss+853vuN++ctfGs1o4K1bt85NnjzZehqmJLndu3dHH1+/ft0Fg0H36quvRtdduXLFBQIB9/rrrxvMcGDcehycc668vNw9/vjjJvOx0tnZ6SS5hoYG59zgPR9uPQ7Opc/5kBZXQlevXtXRo0dVWloas760tFSNjY1Gs7LR0tKi/Px8FRUV6ZlnntHp06etp2SqtbVVHR0dMeeG3+/X3LlzB925IUn19fXKzc3VxIkTtXz5cnV2dlpPKanC4bAkKScnR9LgPR9uPQ43pcP5kBYROn/+vL788kvl5eXFrM/Ly1NHR4fRrAbe9OnTtX37dr377rt644031NHRoZKSEnV1dVlPzczN//6D/dyQpLKyMr355ps6cOCAXnvtNTU3N+vhhx9Wb2+v9dSSwjmnqqoqzZo1S8XFxZIG5/nQ33GQ0ud8SLm7aN/Jrb/awTnXZ10mKysri/550qRJmjlzpr75zW9q27ZtqqqqMpyZvcF+bkjS0qVLo38uLi7W1KlTVVhYqH379mnJkiWGM0uOVatW6aOPPtL777/f57nBdD7c7jiky/mQFldCY8aM0dChQ/v8S6azs7PPv3gGk1GjRmnSpElqaWmxnoqZm+8O5NzoKxQKqbCwMCPPj9WrV2vv3r06ePBgzK9+GWznw+2OQ39S9XxIiwg98MADmjJliurq6mLW19XVqaSkxGhW9np7e/Xpp58qFApZT8VMUVGRgsFgzLlx9epVNTQ0DOpzQ5K6urrU3t6eUeeHc06rVq3Srl27dODAARUVFcU8P1jOh7sdh/6k7Plg+KYIT9566y2XlZXl/vKXv7hPPvnEVVZWulGjRrkzZ85YT23AvPjii66+vt6dPn3aNTU1uccee8xlZ2dn/DHo7u52x44dc8eOHXOS3KZNm9yxY8fcv//9b+ecc6+++qoLBAJu165d7sSJE+7ZZ591oVDIRSIR45kn1p2OQ3d3t3vxxRddY2Oja21tdQcPHnQzZ850X/va1zLqOPz0pz91gUDA1dfXu3PnzkWXS5cuRbcZDOfD3Y5DOp0PaRMh55z74x//6AoLC90DDzzgHnrooZi3Iw4GS5cudaFQyGVlZbn8/Hy3ZMkSd/LkSetpJd3BgwedpD5LeXm5c+7G23LXrVvngsGg8/v9bs6cOe7EiRO2k06COx2HS5cuudLSUjd27FiXlZXlxo8f78rLy11bW5v1tBOqv+9fktu6dWt0m8FwPtztOKTT+cCvcgAAmEmL14QAAJmJCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADDzf266+Tx1oi2RAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "img = xtest[0].squeeze().cpu().numpy()\n",
    "print(img.shape)\n",
    "plt.imshow(img, cmap=\"gray\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ad3bff1",
   "metadata": {},
   "source": [
    "接下来请你选择5个你喜欢的数字（60000以下），然后取出训练集中对应位置的图片，并画出它们的内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "49728eff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZEklEQVR4nO3df2jU9x3H8dfVH9fUXQ5STe6iaQiidDROqLqos/4Cg4HJ1G5YO0pcmWvXRAhpcXP+YRjMdILSP7I61nZZZbr5j7WCrppNk9hZt1R0ZrZzEeNM0ZCZtXcxdSfqZ3+Ix66x0e95l3cu93zAgfe978d7++0Xn/16P+JzzjkBAGDgIesBAADZiwgBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzo60H+KJbt27p0qVLCgQC8vl81uMAADxyzqmvr0+FhYV66KHBr3WGXYQuXbqkoqIi6zEAAA+oq6tLkyZNGnSfYffPcYFAwHoEAEAK3M/f52mL0Ouvv66SkhI9/PDDmjFjho4ePXpf6/gnOAAYGe7n7/O0RGj37t2qqanRxo0bdfLkST311FOqqKjQxYsX0/F0AIAM5UvHt2iXlZXpySef1Pbt2+PbvvrVr2r58uWqr68fdG00GlUwGEz1SACAIRaJRJSbmzvoPim/Erp+/bpOnDih8vLyhO3l5eU6duzYgP1jsZii0WjCDQCQHVIeoStXrujmzZsqKChI2F5QUKDu7u4B+9fX1ysYDMZvvDMOALJH2t6Y8MUXpJxzd32RasOGDYpEIvFbV1dXukYCAAwzKf+c0Pjx4zVq1KgBVz09PT0Dro4kye/3y+/3p3oMAEAGSPmV0NixYzVjxgw1NTUlbG9qatLcuXNT/XQAgAyWlm9MqK2t1XPPPaeZM2dqzpw5+tWvfqWLFy/qxRdfTMfTAQAyVFoitGrVKvX29uqnP/2pLl++rNLSUh04cEDFxcXpeDoAQIZKy+eEHgSfEwKAkcHkc0IAANwvIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABgJuURqqurk8/nS7iFQqFUPw0AYAQYnY7f9IknntAf//jH+P1Ro0al42kAABkuLREaPXo0Vz8AgHtKy2tCHR0dKiwsVElJiZ555hmdP3/+S/eNxWKKRqMJNwBAdkh5hMrKyrRjxw4dPHhQb7zxhrq7uzV37lz19vbedf/6+noFg8H4raioKNUjAQCGKZ9zzqXzCfr7+zV58mStX79etbW1Ax6PxWKKxWLx+9FolBABwAgQiUSUm5s76D5peU3o/40bN07Tpk1TR0fHXR/3+/3y+/3pHgMAMAyl/XNCsVhMH3/8scLhcLqfCgCQYVIeoVdeeUUtLS3q7OzUX/7yF337299WNBpVZWVlqp8KAJDhUv7PcZ988olWr16tK1euaMKECZo9e7aOHz+u4uLiVD8VACDDpf2NCV5Fo1EFg0HrMZDhHn300aTWvfbaa57XfPe73/W85s9//rPnNdXV1Z7X/O1vf/O8BkiV+3ljAt8dBwAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCY4QtMMeyVlZV5XrNt27aknmvOnDlJrfPK5/N5XvPJJ594XrNo0SLPayTp3LlzSa0D/h9fYAoAGNaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABgZrT1AMC9LFiwwPOaofo27GQl8+X1EydO9LzmZz/7mec1kvTcc895XnP9+vWkngvZjSshAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMX2CKYW/mzJlD9lzbt2/3vObWrVue11RVVXlek4zvfOc7Sa1rbGz0vOa9995L6rmQ3bgSAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDM+JxzznqI/xeNRhUMBq3HQJp87Wtf87zmgw8+8Lymp6fH8xpJmjZtmuc148aN87zm8uXLntcMpTfffNPzmh/84AdpmASZLBKJKDc3d9B9uBICAJghQgAAM54j1NraqmXLlqmwsFA+n0979+5NeNw5p7q6OhUWFionJ0cLFy7UmTNnUjUvAGAE8Ryh/v5+TZ8+XQ0NDXd9fMuWLdq2bZsaGhrU1tamUCikJUuWqK+v74GHBQCMLJ5/smpFRYUqKiru+phzTq+99po2btyolStXSpLefvttFRQUaNeuXXrhhRcebFoAwIiS0teEOjs71d3drfLy8vg2v9+vBQsW6NixY3ddE4vFFI1GE24AgOyQ0gh1d3dLkgoKChK2FxQUxB/7ovr6egWDwfitqKgolSMBAIaxtLw7zufzJdx3zg3YdseGDRsUiUTit66urnSMBAAYhjy/JjSYUCgk6fYVUTgcjm/v6ekZcHV0h9/vl9/vT+UYAIAMkdIroZKSEoVCITU1NcW3Xb9+XS0tLZo7d24qnwoAMAJ4vhK6evWqzp07F7/f2dmpU6dOKS8vT4899phqamq0efNmTZkyRVOmTNHmzZv1yCOP6Nlnn03p4ACAzOc5Qh9++KEWLVoUv19bWytJqqys1G9+8xutX79e165d00svvaRPP/1UZWVlOnTokAKBQOqmBgCMCHyBKYbUW2+95XnN888/73nNu+++63mNJC1fvjypdV5973vf87wmmWOXrGvXrnleU1ZW5nnN3//+d89rkDn4AlMAwLBGhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAMyn9yarAvbS3t3tek8wXvU+ePNnzGknKycnxvCaZb5z+6KOPPK+5efOm5zWjRo3yvAYYSlwJAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmfC6Zb4dMo2g0qmAwaD0G0uTxxx/3vOavf/2r5zVf+cpXPK+RpNWrV3tes3v3bs9rHn30Uc9rTp065XnNxIkTPa+RpEOHDnles3Tp0qSeCyNXJBJRbm7uoPtwJQQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmBltPQCyyz/+8Q/Pa3796197XvP88897XiNJb775puc1yXxx54oVKzyvudcXQabSn/70pyF7LmQ3roQAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADN8gSmGvZqaGs9r/vCHPyT1XN///vc9r6msrEzquYazf/7zn9YjIEtwJQQAMEOEAABmPEeotbVVy5YtU2FhoXw+n/bu3Zvw+Jo1a+Tz+RJus2fPTtW8AIARxHOE+vv7NX36dDU0NHzpPkuXLtXly5fjtwMHDjzQkACAkcnzGxMqKipUUVEx6D5+v1+hUCjpoQAA2SEtrwk1NzcrPz9fU6dO1dq1a9XT0/Ol+8ZiMUWj0YQbACA7pDxCFRUV2rlzpw4fPqytW7eqra1NixcvViwWu+v+9fX1CgaD8VtRUVGqRwIADFMp/5zQqlWr4r8uLS3VzJkzVVxcrP3792vlypUD9t+wYYNqa2vj96PRKCECgCyR9g+rhsNhFRcXq6Oj466P+/1++f3+dI8BABiG0v45od7eXnV1dSkcDqf7qQAAGcbzldDVq1d17ty5+P3Ozk6dOnVKeXl5ysvLU11dnZ5++mmFw2FduHBBP/nJTzR+/HitWLEipYMDADKf5wh9+OGHWrRoUfz+nddzKisrtX37drW3t2vHjh367LPPFA6HtWjRIu3evVuBQCB1UwMARgSfc85ZD/H/otGogsGg9RhAWs2bN8/zmtbW1jRMcncTJkzwvKa3tzcNkyCTRSIR5ebmDroP3x0HADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM2n/yaoABhpmX14PmOFKCABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGDGU4Tq6+s1a9YsBQIB5efna/ny5Tp79mzCPs451dXVqbCwUDk5OVq4cKHOnDmT0qEBACODpwi1tLSoqqpKx48fV1NTk27cuKHy8nL19/fH99myZYu2bdumhoYGtbW1KRQKacmSJerr60v58ACAzOZzzrlkF//73/9Wfn6+WlpaNH/+fDnnVFhYqJqaGv3oRz+SJMViMRUUFOjnP/+5XnjhhXv+ntFoVMFgMNmRgIzwjW98w/Oao0ePpmGSu5swYYLnNb29vWmYBJksEokoNzd30H0e6DWhSCQiScrLy5MkdXZ2qru7W+Xl5fF9/H6/FixYoGPHjt3194jFYopGowk3AEB2SDpCzjnV1tZq3rx5Ki0tlSR1d3dLkgoKChL2LSgoiD/2RfX19QoGg/FbUVFRsiMBADJM0hGqrq7W6dOn9bvf/W7AYz6fL+G+c27Atjs2bNigSCQSv3V1dSU7EgAgw4xOZtG6deu0b98+tba2atKkSfHtoVBI0u0ronA4HN/e09Mz4OroDr/fL7/fn8wYAIAM5+lKyDmn6upq7dmzR4cPH1ZJSUnC4yUlJQqFQmpqaopvu379ulpaWjR37tzUTAwAGDE8XQlVVVVp165devfddxUIBOKv8wSDQeXk5Mjn86mmpkabN2/WlClTNGXKFG3evFmPPPKInn322bT8AQAAmctThLZv3y5JWrhwYcL2xsZGrVmzRpK0fv16Xbt2TS+99JI+/fRTlZWV6dChQwoEAikZGAAwcjzQ54TSgc8JIRvMmzfP85rW1lbPa/7zn/94XiNJkydP9rzmzkc2gDvS/jkhAAAeBBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwk9ZNVATyYadOmDcnzHD16NKl1fCM2hgpXQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGb7AFDAwY8YM6xGAYYErIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAmdHWAwDZqLGx0fOap59+Og2TALa4EgIAmCFCAAAzniJUX1+vWbNmKRAIKD8/X8uXL9fZs2cT9lmzZo18Pl/Cbfbs2SkdGgAwMniKUEtLi6qqqnT8+HE1NTXpxo0bKi8vV39/f8J+S5cu1eXLl+O3AwcOpHRoAMDI4OmNCe+9917C/cbGRuXn5+vEiROaP39+fLvf71coFErNhACAEeuBXhOKRCKSpLy8vITtzc3Nys/P19SpU7V27Vr19PR86e8Ri8UUjUYTbgCA7JB0hJxzqq2t1bx581RaWhrfXlFRoZ07d+rw4cPaunWr2tratHjxYsVisbv+PvX19QoGg/FbUVFRsiMBADJM0p8Tqq6u1unTp/X+++8nbF+1alX816WlpZo5c6aKi4u1f/9+rVy5csDvs2HDBtXW1sbvR6NRQgQAWSKpCK1bt0779u1Ta2urJk2aNOi+4XBYxcXF6ujouOvjfr9ffr8/mTEAABnOU4Scc1q3bp3eeecdNTc3q6Sk5J5rent71dXVpXA4nPSQAICRydNrQlVVVfrtb3+rXbt2KRAIqLu7W93d3bp27Zok6erVq3rllVf0wQcf6MKFC2pubtayZcs0fvx4rVixIi1/AABA5vJ0JbR9+3ZJ0sKFCxO2NzY2as2aNRo1apTa29u1Y8cOffbZZwqHw1q0aJF2796tQCCQsqEBACOD53+OG0xOTo4OHjz4QAMBALKHz92rLEMsGo0qGAxajwEAeECRSES5ubmD7sMXmAIAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGBm2EXIOWc9AgAgBe7n7/NhF6G+vj7rEQAAKXA/f5/73DC79Lh165YuXbqkQCAgn8+X8Fg0GlVRUZG6urqUm5trNKE9jsNtHIfbOA63cRxuGw7HwTmnvr4+FRYW6qGHBr/WGT1EM923hx56SJMmTRp0n9zc3Kw+ye7gONzGcbiN43Abx+E26+MQDAbva79h989xAIDsQYQAAGYyKkJ+v1+bNm2S3++3HsUUx+E2jsNtHIfbOA63ZdpxGHZvTAAAZI+MuhICAIwsRAgAYIYIAQDMECEAgJmMitDrr7+ukpISPfzww5oxY4aOHj1qPdKQqqurk8/nS7iFQiHrsdKutbVVy5YtU2FhoXw+n/bu3ZvwuHNOdXV1KiwsVE5OjhYuXKgzZ87YDJtG9zoOa9asGXB+zJ4922bYNKmvr9esWbMUCASUn5+v5cuX6+zZswn7ZMP5cD/HIVPOh4yJ0O7du1VTU6ONGzfq5MmTeuqpp1RRUaGLFy9ajzaknnjiCV2+fDl+a29vtx4p7fr7+zV9+nQ1NDTc9fEtW7Zo27ZtamhoUFtbm0KhkJYsWTLivofwXsdBkpYuXZpwfhw4cGAIJ0y/lpYWVVVV6fjx42pqatKNGzdUXl6u/v7++D7ZcD7cz3GQMuR8cBni61//unvxxRcTtj3++OPuxz/+sdFEQ2/Tpk1u+vTp1mOYkuTeeeed+P1bt265UCjkXn311fi2//73vy4YDLpf/vKXBhMOjS8eB+ecq6ysdN/61rdM5rHS09PjJLmWlhbnXPaeD188Ds5lzvmQEVdC169f14kTJ1ReXp6wvby8XMeOHTOaykZHR4cKCwtVUlKiZ555RufPn7ceyVRnZ6e6u7sTzg2/368FCxZk3bkhSc3NzcrPz9fUqVO1du1a9fT0WI+UVpFIRJKUl5cnKXvPhy8ehzsy4XzIiAhduXJFN2/eVEFBQcL2goICdXd3G0019MrKyrRjxw4dPHhQb7zxhrq7uzV37lz19vZaj2bmzn//bD83JKmiokI7d+7U4cOHtXXrVrW1tWnx4sWKxWLWo6WFc061tbWaN2+eSktLJWXn+XC34yBlzvkw7L5FezBf/NEOzrkB20ayioqK+K+nTZumOXPmaPLkyXr77bdVW1trOJm9bD83JGnVqlXxX5eWlmrmzJkqLi7W/v37tXLlSsPJ0qO6ulqnT5/W+++/P+CxbDofvuw4ZMr5kBFXQuPHj9eoUaMG/J9MT0/PgP/jySbjxo3TtGnT1NHRYT2KmTvvDuTcGCgcDqu4uHhEnh/r1q3Tvn37dOTIkYQf/ZJt58OXHYe7Ga7nQ0ZEaOzYsZoxY4aampoStjc1NWnu3LlGU9mLxWL6+OOPFQ6HrUcxU1JSolAolHBuXL9+XS0tLVl9bkhSb2+vurq6RtT54ZxTdXW19uzZo8OHD6ukpCTh8Ww5H+51HO5m2J4Phm+K8OT3v/+9GzNmjHvrrbfcRx995Gpqaty4cePchQsXrEcbMi+//LJrbm5258+fd8ePH3ff/OY3XSAQGPHHoK+vz508edKdPHnSSXLbtm1zJ0+edP/617+cc869+uqrLhgMuj179rj29na3evVqFw6HXTQaNZ48tQY7Dn19fe7ll192x44dc52dne7IkSNuzpw5buLEiSPqOPzwhz90wWDQNTc3u8uXL8dvn3/+eXyfbDgf7nUcMul8yJgIOefcL37xC1dcXOzGjh3rnnzyyYS3I2aDVatWuXA47MaMGeMKCwvdypUr3ZkzZ6zHSrsjR444SQNulZWVzrnbb8vdtGmTC4VCzu/3u/nz57v29nbbodNgsOPw+eefu/LycjdhwgQ3ZswY99hjj7nKykp38eJF67FT6m5/fkmusbExvk82nA/3Og6ZdD7woxwAAGYy4jUhAMDIRIQAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCY+R9oNKrbEjdRkQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZtklEQVR4nO3df0xV9/3H8dfV6q06uAtBuJeJfMmmm6nGpepUYv3RTCpbXa1dZtulwf7h7PyRMNp0s2aR7Q/pWGrWhdW1ZnGa1dVkWueimWVRwMbZqKGt09bQiMIihEncvYgWZvl8/yC96VVEz/Ve3tzL85F8Ejn3vDlvPh55+bn33HN9zjknAAAMjLBuAAAwfBFCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMHOfdQM36+3t1aVLl5SRkSGfz2fdDgDAI+ecOjs7lZeXpxEjBl7rDLkQunTpkvLz863bAADco5aWFk2YMGHAfYbc03EZGRnWLQAAEuBufp8nLYRee+01FRYW6v7779eMGTN09OjRu6rjKTgASA938/s8KSG0e/dulZWVaePGjWpoaNBDDz2kkpISNTc3J+NwAIAU5UvGXbRnz56tBx98UFu3bo1umzJlipYtW6bKysoBayORiAKBQKJbAgAMsnA4rMzMzAH3SfhKqKenR6dOnVJxcXHM9uLiYh07duyW/bu7uxWJRGIGAGB4SHgIXb58WZ999plyc3Njtufm5qqtre2W/SsrKxUIBKKDK+MAYPhI2oUJN78g5Zzr90WqDRs2KBwOR0dLS0uyWgIADDEJf59Qdna2Ro4cecuqp729/ZbVkST5/X75/f5EtwEASAEJXwmNHj1aM2bMUE1NTcz2mpoaFRUVJfpwAIAUlpQ7JpSXl+uZZ57RzJkzNXfuXL3xxhtqbm7Wc889l4zDAQBSVFJCaMWKFero6NAvf/lLtba2aurUqTp48KAKCgqScTgAQIpKyvuE7gXvEwKA9GDyPiEAAO4WIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMHOfdQMA7s6Xv/xlzzXPPPNMXMf67W9/67nmX//6l+eaRx991HPNxYsXPddg6GIlBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAw3MAVSxKuvvuq55oc//GFcx+rt7fVck52dPSg13MA0vbASAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYbmAIGZs6c6bnmkUceSUIniXPy5EnPNadOnUpCJ0glrIQAAGYIIQCAmYSHUEVFhXw+X8wIBoOJPgwAIA0k5TWhBx54QP/4xz+iX48cOTIZhwEApLikhNB9993H6gcAcEdJeU2osbFReXl5Kiws1JNPPqnz58/fdt/u7m5FIpGYAQAYHhIeQrNnz9bOnTt16NAhbdu2TW1tbSoqKlJHR0e/+1dWVioQCERHfn5+olsCAAxRCQ+hkpISPfHEE5o2bZq+/e1v68CBA5KkHTt29Lv/hg0bFA6Ho6OlpSXRLQEAhqikv1l13LhxmjZtmhobG/t93O/3y+/3J7sNAMAQlPT3CXV3d+ujjz5SKBRK9qEAACkm4SH0wgsvqK6uTk1NTXrvvff0/e9/X5FIRKWlpYk+FAAgxSX86bh///vfeuqpp3T58mWNHz9ec+bM0fHjx1VQUJDoQwEAUlzCQ+itt95K9LcEhjSfz+e5ZvXq1Z5rsrOzPdcMpr1791q3gBTEveMAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCYSfqH2gHpbt26dZ5rnn322SR0kjg/+MEPPNfs2bMnCZ0g3bESAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCY4S7awD2aMmWKdQu3dfbs2bjquCM2BgsrIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCYIYQAAGa4gSnS0ujRo+OqKy8v91yzevVqzzXOOc81ly5d8lxTVVXluQYYTKyEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmPG5eO6kmESRSESBQMC6DaS4r33ta3HVffzxx55rfD6f55p4/tl997vf9Vxz6NAhzzVAooTDYWVmZg64DyshAIAZQggAYMZzCNXX12vp0qXKy8uTz+fTvn37Yh53zqmiokJ5eXkaM2aMFi5cqDNnziSqXwBAGvEcQl1dXZo+fbqqq6v7fbyqqkpbtmxRdXW1Tpw4oWAwqMWLF6uzs/OemwUApBfPn6xaUlKikpKSfh9zzuk3v/mNNm7cqOXLl0uSduzYodzcXO3atSuuT6AEAKSvhL4m1NTUpLa2NhUXF0e3+f1+LViwQMeOHeu3pru7W5FIJGYAAIaHhIZQW1ubJCk3Nzdme25ubvSxm1VWVioQCERHfn5+IlsCAAxhSbk67ub3TTjnbvteig0bNigcDkdHS0tLMloCAAxBnl8TGkgwGJTUtyIKhULR7e3t7besjj7n9/vl9/sT2QYAIEUkdCVUWFioYDCompqa6Laenh7V1dWpqKgokYcCAKQBzyuhq1ev6pNPPol+3dTUpPfff19ZWVmaOHGiysrKtHnzZk2aNEmTJk3S5s2bNXbsWD399NMJbRwAkPo8h9DJkye1aNGi6Nfl5eWSpNLSUv3xj3/Uiy++qOvXr2vNmjW6cuWKZs+erXfeeUcZGRmJ6xoAkBa4gSnS0sGDB+Oqe+SRRzzXjBjh/VntpqYmzzXf+c53PNfEc0NWIFG4gSkAYEgjhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJhJ6CerAsmwYMECzzXz5s2L61jx3FS+t7fXc81f/vIXzzUtLS2ea0pKSjzXSNKrr77quSaeuduyZYvnmtdff91zDYYuVkIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMcANTDHk/+clPPNeMHTs2CZ30b//+/Z5rfv3rXw9KzY9+9CPPNZLk8/k818RzA9OqqirPNa2trZ5r4vk7wuBgJQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMNzDFkDdlyhTrFgZ08eJFzzWbN2/2XPPss896rhnqxo0b57lm4sSJSegEVlgJAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMMMNTDHk+Xy+QamJ1ze/+U3PNfHclHUwf6YRI7z//7S3tzcJnSDdsRICAJghhAAAZjyHUH19vZYuXaq8vDz5fD7t27cv5vGVK1fK5/PFjDlz5iSqXwBAGvEcQl1dXZo+fbqqq6tvu8+SJUvU2toaHQcPHrynJgEA6cnzhQklJSUqKSkZcB+/369gMBh3UwCA4SEprwnV1tYqJydHkydP1qpVq9Te3n7bfbu7uxWJRGIGAGB4SHgIlZSU6M0339Thw4f1yiuv6MSJE3r44YfV3d3d7/6VlZUKBALRkZ+fn+iWAABDVMLfJ7RixYron6dOnaqZM2eqoKBABw4c0PLly2/Zf8OGDSovL49+HYlECCIAGCaS/mbVUCikgoICNTY29vu43++X3+9PdhsAgCEo6e8T6ujoUEtLi0KhULIPBQBIMZ5XQlevXtUnn3wS/bqpqUnvv/++srKylJWVpYqKCj3xxBMKhUK6cOGCXnrpJWVnZ+vxxx9PaOMAgNTnOYROnjypRYsWRb/+/PWc0tJSbd26VadPn9bOnTv13//+V6FQSIsWLdLu3buVkZGRuK4BAGnB55xz1k18USQSUSAQsG4DSfJ///d/nmuOHj3quWYwn/6N58aig/XP7uzZs3HVffzxx55r+rvw6E4uXLjguWbGjBmea8LhsOca3LtwOKzMzMwB9+HecQAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM0n/ZFXgi5YuXeq5hg9E7PPBBx94rlm9enVcx3rvvfc811y9etVzTVVVleca7oidXlgJAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMMMNTIEU0dzc7LnmjTfeSEIn/auvr/dcM5j9YWhiJQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMNzDFoPL5fINSM5hGjPD+f7ne3l7PNd/73vc818Qrnp/p3XffTUInSHeshAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJjxOeecdRNfFIlEFAgErNtAksTzd9vQ0OC5ZuLEiZ5r4hXPDVaH2D+7W3z961/3XNPc3Oy55n//+5/nGqSOcDiszMzMAfdhJQQAMEMIAQDMeAqhyspKzZo1SxkZGcrJydGyZct07ty5mH2cc6qoqFBeXp7GjBmjhQsX6syZMwltGgCQHjyFUF1dndauXavjx4+rpqZGN27cUHFxsbq6uqL7VFVVacuWLaqurtaJEycUDAa1ePFidXZ2Jrx5AEBqu6cLE/7zn/8oJydHdXV1mj9/vpxzysvLU1lZmX76059Kkrq7u5Wbm6tf/epXWr169R2/JxcmpDcuTOjDhQl9uDAhvSX9woRwOCxJysrKkiQ1NTWpra1NxcXF0X38fr8WLFigY8eO9fs9uru7FYlEYgYAYHiIO4SccyovL9e8efM0depUSVJbW5skKTc3N2bf3Nzc6GM3q6ysVCAQiI78/Px4WwIApJi4Q2jdunX68MMP9ec///mWx25+esI5d9unLDZs2KBwOBwdLS0t8bYEAEgx98VTtH79eu3fv1/19fWaMGFCdHswGJTUtyIKhULR7e3t7besjj7n9/vl9/vjaQMAkOI8rYScc1q3bp327t2rw4cPq7CwMObxwsJCBYNB1dTURLf19PSorq5ORUVFiekYAJA2PK2E1q5dq127dumvf/2rMjIyoq/zBAIBjRkzRj6fT2VlZdq8ebMmTZqkSZMmafPmzRo7dqyefvrppPwAAIDU5SmEtm7dKklauHBhzPbt27dr5cqVkqQXX3xR169f15o1a3TlyhXNnj1b77zzjjIyMhLSMAAgfXADUwx5d/P+sptVVVXFdaxx48Z5rhms9wl98MEHnmteeuklzzWSdOjQobjqgC/iBqYAgCGNEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGAmrk9WBQbT66+/7rmmp6cnrmNt27bNc019fb3nmj179niu+dvf/ua55uLFi55rgMHESggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZn3POWTfxRZFIRIFAwLoNAMA9CofDyszMHHAfVkIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzHgKocrKSs2aNUsZGRnKycnRsmXLdO7cuZh9Vq5cKZ/PFzPmzJmT0KYBAOnBUwjV1dVp7dq1On78uGpqanTjxg0VFxerq6srZr8lS5aotbU1Og4ePJjQpgEA6eE+Lzv//e9/j/l6+/btysnJ0alTpzR//vzodr/fr2AwmJgOAQBp655eEwqHw5KkrKysmO21tbXKycnR5MmTtWrVKrW3t9/2e3R3dysSicQMAMDw4HPOuXgKnXN67LHHdOXKFR09ejS6fffu3frSl76kgoICNTU16ec//7lu3LihU6dOye/33/J9Kioq9Itf/CL+nwAAMCSFw2FlZmYOvJOL05o1a1xBQYFraWkZcL9Lly65UaNGuT179vT7+KeffurC4XB0tLS0OEkMBoPBSPERDofvmCWeXhP63Pr167V//37V19drwoQJA+4bCoVUUFCgxsbGfh/3+/39rpAAAOnPUwg557R+/Xq9/fbbqq2tVWFh4R1rOjo61NLSolAoFHeTAID05OnChLVr1+pPf/qTdu3apYyMDLW1tamtrU3Xr1+XJF29elUvvPCC/vnPf+rChQuqra3V0qVLlZ2drccffzwpPwAAIIV5eR1It3neb/v27c45565du+aKi4vd+PHj3ahRo9zEiRNdaWmpa25uvutjhMNh8+cxGQwGg3Hv425eE4r76rhkiUQiCgQC1m0AAO7R3Vwdx73jAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmhlwIOeesWwAAJMDd/D4fciHU2dlp3QIAIAHu5ve5zw2xpUdvb68uXbqkjIwM+Xy+mMcikYjy8/PV0tKizMxMow7tMQ99mIc+zEMf5qHPUJgH55w6OzuVl5enESMGXuvcN0g93bURI0ZowoQJA+6TmZk5rE+yzzEPfZiHPsxDH+ahj/U8BAKBu9pvyD0dBwAYPgghAICZlAohv9+vTZs2ye/3W7diinnowzz0YR76MA99Um0ehtyFCQCA4SOlVkIAgPRCCAEAzBBCAAAzhBAAwExKhdBrr72mwsJC3X///ZoxY4aOHj1q3dKgqqiokM/nixnBYNC6raSrr6/X0qVLlZeXJ5/Pp3379sU87pxTRUWF8vLyNGbMGC1cuFBnzpyxaTaJ7jQPK1euvOX8mDNnjk2zSVJZWalZs2YpIyNDOTk5WrZsmc6dOxezz3A4H+5mHlLlfEiZENq9e7fKysq0ceNGNTQ06KGHHlJJSYmam5utWxtUDzzwgFpbW6Pj9OnT1i0lXVdXl6ZPn67q6up+H6+qqtKWLVtUXV2tEydOKBgMavHixWl3H8I7zYMkLVmyJOb8OHjw4CB2mHx1dXVau3atjh8/rpqaGt24cUPFxcXq6uqK7jMczoe7mQcpRc4HlyK+9a1vueeeey5m2ze+8Q33s5/9zKijwbdp0yY3ffp06zZMSXJvv/129Ove3l4XDAbdyy+/HN326aefukAg4H7/+98bdDg4bp4H55wrLS11jz32mEk/Vtrb250kV1dX55wbvufDzfPgXOqcDymxEurp6dGpU6dUXFwcs724uFjHjh0z6spGY2Oj8vLyVFhYqCeffFLnz5+3bslUU1OT2traYs4Nv9+vBQsWDLtzQ5Jqa2uVk5OjyZMna9WqVWpvb7duKanC4bAkKSsrS9LwPR9unofPpcL5kBIhdPnyZX322WfKzc2N2Z6bm6u2tjajrgbf7NmztXPnTh06dEjbtm1TW1ubioqK1NHRYd2amc///of7uSFJJSUlevPNN3X48GG98sorOnHihB5++GF1d3dbt5YUzjmVl5dr3rx5mjp1qqTheT70Nw9S6pwPQ+4u2gO5+aMdnHO3bEtnJSUl0T9PmzZNc+fO1Ve/+lXt2LFD5eXlhp3ZG+7nhiStWLEi+uepU6dq5syZKigo0IEDB7R8+XLDzpJj3bp1+vDDD/Xuu+/e8thwOh9uNw+pcj6kxEooOztbI0eOvOV/Mu3t7bf8j2c4GTdunKZNm6bGxkbrVsx8fnUg58atQqGQCgoK0vL8WL9+vfbv368jR47EfPTLcDsfbjcP/Rmq50NKhNDo0aM1Y8YM1dTUxGyvqalRUVGRUVf2uru79dFHHykUClm3YqawsFDBYDDm3Ojp6VFdXd2wPjckqaOjQy0tLWl1fjjntG7dOu3du1eHDx9WYWFhzOPD5Xy40zz0Z8ieD4YXRXjy1ltvuVGjRrk//OEP7uzZs66srMyNGzfOXbhwwbq1QfP888+72tpad/78eXf8+HH36KOPuoyMjLSfg87OTtfQ0OAaGhqcJLdlyxbX0NDgLl686Jxz7uWXX3aBQMDt3bvXnT592j311FMuFAq5SCRi3HliDTQPnZ2d7vnnn3fHjh1zTU1N7siRI27u3LnuK1/5SlrNw49//GMXCARcbW2ta21tjY5r165F9xkO58Od5iGVzoeUCSHnnPvd737nCgoK3OjRo92DDz4YcznicLBixQoXCoXcqFGjXF5enlu+fLk7c+aMdVtJd+TIESfpllFaWuqc67ssd9OmTS4YDDq/3+/mz5/vTp8+bdt0Egw0D9euXXPFxcVu/PjxbtSoUW7ixImutLTUNTc3W7edUP39/JLc9u3bo/sMh/PhTvOQSucDH+UAADCTEq8JAQDSEyEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADP/Dytn71+SMsNjAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAa90lEQVR4nO3df2xV9f3H8ddV6hXx9i4V23sr2DUKcxHGIiI/pogGGppIQFyGuriSOCc/N1acGSOGumXUkcj8blW3kcFgAyQZ6khgaBdsYWFdkKEQRFJnlW5w04Hs3vKriH6+fxBudm2pfC739t17+3wkn8R7znlz3hxP+uLTc+/nBpxzTgAAGLjCugEAQN9FCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMBMP+sGPuvTTz/V4cOHFQqFFAgErNsBAHhyzqm9vV2lpaW64oru5zq9LoQOHz6swYMHW7cBALhMra2tGjRoULfH9Lpfx4VCIesWAAAZcCk/z7MWQi+88ILKy8t19dVXa+TIkdqxY8cl1fErOADID5fy8zwrIbRhwwYtWLBAixcv1p49e3TXXXepsrJShw4dysbpAAA5KpCNVbRHjx6t2267TS+++GJy25e//GVNmzZNtbW13dYmEgmFw+FMtwQA6GHxeFyFhYXdHpPxmdDZs2e1e/duVVRUpGyvqKjQzp07Ox3f0dGhRCKRMgAAfUPGQ+jo0aP65JNPVFJSkrK9pKREsVis0/G1tbUKh8PJwTvjAKDvyNobEz77QMo51+VDqkWLFikejydHa2trtloCAPQyGf+c0MCBA3XllVd2mvW0tbV1mh1JUjAYVDAYzHQbAIAckPGZ0FVXXaWRI0eqvr4+ZXt9fb3GjRuX6dMBAHJYVlZMqK6u1iOPPKLbb79dY8eO1W9+8xsdOnRIs2bNysbpAAA5KishNGPGDB07dkw//vGPdeTIEQ0bNkxbtmxRWVlZNk4HAMhRWfmc0OXgc0IAkB9MPicEAMClIoQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAmX7WDQC5LhAIeNdMnDjRuyYSiXjX/P73v/eukaSbb77Zu+baa6/1rqmsrPSu+dKXvuRdM2rUKO8aSfr444+9a2bNmuVd09TU5F2TL5gJAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMMMCpsBl+sY3vuFds379eu+a7du3e9dcd9113jWS9PTTT3vXhEKhtM6Vb6ZOnepdwwKmAAAYIIQAAGYyHkI1NTUKBAIpI53vQQEA5L+sPBO69dZb9Ze//CX5+sorr8zGaQAAOS4rIdSvXz9mPwCAz5WVZ0LNzc0qLS1VeXm5HnzwQb3//vsXPbajo0OJRCJlAAD6hoyH0OjRo7VmzRq99tprWrFihWKxmMaNG6djx451eXxtba3C4XByDB48ONMtAQB6qYyHUGVlpR544AENHz5cEydO1ObNmyVJq1ev7vL4RYsWKR6PJ0dra2umWwIA9FJZ/7DqgAEDNHz4cDU3N3e5PxgMKhgMZrsNAEAvlPXPCXV0dOjAgQOKRqPZPhUAIMdkPISeeOIJNTY2qqWlRX//+9/19a9/XYlEQlVVVZk+FQAgx2X813H/+te/9NBDD+no0aO6/vrrNWbMGDU1NamsrCzTpwIA5LiMh9BLL72U6T8S6DHpfL4tnQVM0zF+/PgeqUnXxd4B2509e/Z41+zdu9e7prq62rsmXQcOHOixc+UD1o4DAJghhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABgJutfagdY+MpXvpJW3WuvveZdU1JSkta5esLbb7+dVt3y5cu9a3bs2OFdc/bsWe+arVu3etek62JfxtmdDz/8MAud5C9mQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM6yijR4VjUa9a2bPnu1d853vfMe7RpKKi4vTqvOVzurWP/3pT71rNm/e7F0jSadPn/auueGGG7xr0lm1/NZbb/WuSWe1bkmaOnWqd827776b1rn6KmZCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzLCAKdL2ta99zbtm5cqV3jVDhw71rnHOedeka9OmTd4106ZNy3wjGTRmzBjvmhUrVnjXpLMY6ccff+xd8+ijj3rXSCxG2hOYCQEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADDDAqZQJBJJq66urs67ZsiQId41PbkY6S9/+UvvmqeffjoLnWTGfffdl1bdL37xC++aL37xi941586d867585//7F2zdu1a7xr0DGZCAAAzhBAAwIx3CG3fvl1TpkxRaWmpAoGAXn311ZT9zjnV1NSotLRU/fv314QJE7R///5M9QsAyCPeIXTy5EmNGDHios8Dli1bpuXLl6uurk67du1SJBLRpEmT1N7eftnNAgDyi/cbEyorK1VZWdnlPuecnnvuOS1evFjTp0+XJK1evVolJSVat26dHn/88cvrFgCQVzL6TKilpUWxWEwVFRXJbcFgUHfffbd27tzZZU1HR4cSiUTKAAD0DRkNoVgsJkkqKSlJ2V5SUpLc91m1tbUKh8PJMXjw4Ey2BADoxbLy7rhAIJDy2jnXadsFixYtUjweT47W1tZstAQA6IUy+mHVCx96jMViikajye1tbW2dZkcXBINBBYPBTLYBAMgRGZ0JlZeXKxKJqL6+Prnt7Nmzamxs1Lhx4zJ5KgBAHvCeCZ04cULvvfde8nVLS4veeustFRUV6cYbb9SCBQu0dOlSDRkyREOGDNHSpUt1zTXX6OGHH85o4wCA3OcdQm+++abuueee5Ovq6mpJUlVVlX73u9/pySef1OnTpzVnzhwdP35co0eP1uuvv65QKJS5rgEAeSHgenJ1yEuQSCQUDoet2+hTnn322bTqvv/972e4k8yZPXt2WnW//vWvM9xJ19L5R9k3v/lN75rnn3/eu0bq/OaiS9HR0eFd8+1vf9u7hsVIc0c8HldhYWG3x7B2HADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADKtoQ0OHDk2rbuvWrd411113nXfNT37yE++a//u///OukaSPP/7Yu6agoMC7ZunSpd41Cxcu9K5JVzo/Fr71rW9517Aidn5jFW0AQK9GCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADAuYokfdcsst3jXvvvtuFjrpWigU8q7ZsGGDd83kyZO9a9Jx5syZtOoeffRR75r169endS7kLxYwBQD0aoQQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMz0s24AfUtPLUYaDAbTqlu3bp13TU8tRpqOsWPHplX39ttvZ7gToGvMhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJhhAVP0enfeead3zaZNm9I61xe+8IW06nyls0DoD37wA++avXv3etcAPYmZEADADCEEADDjHULbt2/XlClTVFpaqkAgoFdffTVl/8yZMxUIBFLGmDFjMtUvACCPeIfQyZMnNWLECNXV1V30mMmTJ+vIkSPJsWXLlstqEgCQn7zfmFBZWanKyspujwkGg4pEImk3BQDoG7LyTKihoUHFxcUaOnSoHnvsMbW1tV302I6ODiUSiZQBAOgbMh5ClZWVWrt2rbZt26Znn31Wu3bt0r333quOjo4uj6+trVU4HE6OwYMHZ7olAEAvlfHPCc2YMSP538OGDdPtt9+usrIybd68WdOnT+90/KJFi1RdXZ18nUgkCCIA6COy/mHVaDSqsrIyNTc3d7k/GAwqGAxmuw0AQC+U9c8JHTt2TK2trYpGo9k+FQAgx3jPhE6cOKH33nsv+bqlpUVvvfWWioqKVFRUpJqaGj3wwAOKRqP64IMP9KMf/UgDBw7U/fffn9HGAQC5zzuE3nzzTd1zzz3J1xee51RVVenFF1/Uvn37tGbNGv33v/9VNBrVPffcow0bNigUCmWuawBAXgg455x1E/8rkUgoHA5bt4EseeSRR7xrfv7zn3vXFBUVedeka+PGjd418+fP966JxWLeNYCleDyuwsLCbo9h7TgAgBlCCABghhACAJghhAAAZgghAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgJmsf7Mqer+CgoK06n72s59513zve9/zrgkEAt416S4Ov3LlSu+aJUuWeNewIjZwHjMhAIAZQggAYIYQAgCYIYQAAGYIIQCAGUIIAGCGEAIAmCGEAABmCCEAgBlCCABghhACAJghhAAAZljAFFq2bFladeksRtpTli5dmlbdU089leFOAHSHmRAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzLGCaZ0aPHu1dM3fu3Cx00rUTJ05411RUVHjX7Nq1y7smH1177bVp1fXv39+75uTJk941p06d8q5BfmEmBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwLmPZioVDIu2blypXeNf369dxt8Mc//tG7pqmpKQud2CosLPSuueOOO7xrXnjhBe8aSbr55pu9a+rq6rxrvvvd73rXIL8wEwIAmCGEAABmvEKotrZWo0aNUigUUnFxsaZNm6aDBw+mHOOcU01NjUpLS9W/f39NmDBB+/fvz2jTAID84BVCjY2Nmjt3rpqamlRfX69z586poqIi5cusli1bpuXLl6uurk67du1SJBLRpEmT1N7envHmAQC5zeuJ9NatW1Ner1q1SsXFxdq9e7fGjx8v55yee+45LV68WNOnT5ckrV69WiUlJVq3bp0ef/zxzHUOAMh5l/VMKB6PS5KKiookSS0tLYrFYilfxxwMBnX33Xdr586dXf4ZHR0dSiQSKQMA0DekHULOOVVXV+vOO+/UsGHDJEmxWEySVFJSknJsSUlJct9n1dbWKhwOJ8fgwYPTbQkAkGPSDqF58+Zp7969Wr9+fad9gUAg5bVzrtO2CxYtWqR4PJ4cra2t6bYEAMgxaX1Kcf78+dq0aZO2b9+uQYMGJbdHIhFJ52dE0Wg0ub2tra3T7OiCYDCoYDCYThsAgBznNRNyzmnevHl6+eWXtW3bNpWXl6fsLy8vVyQSUX19fXLb2bNn1djYqHHjxmWmYwBA3vCaCc2dO1fr1q3Tn/70J4VCoeRznnA4rP79+ysQCGjBggVaunSphgwZoiFDhmjp0qW65ppr9PDDD2flLwAAyF1eIfTiiy9KkiZMmJCyfdWqVZo5c6Yk6cknn9Tp06c1Z84cHT9+XKNHj9brr7+e1jpoAID8FnDOOesm/lcikVA4HLZuo1f46le/6l3zj3/8I/ONZNBHH33kXXP8+PEsdGIrneeg//v8tTc6d+6cd82kSZO8axobG71rYCMej3/uYr2sHQcAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMJPWN6uiZxw6dMi75p///Kd3zU033eRdk66ioqIeqUHPO3jwoHfNe++9l4VOkEuYCQEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADDDAqa92EcffeRdU1lZ6V0zceJE7xpJWrx4sXfNDTfckNa58s1//vMf75qNGzdmoZOuvfPOO941zc3N3jX//ve/vWuQX5gJAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMBNwzjnrJv5XIpFQOBy2bgMAcJni8bgKCwu7PYaZEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzHiFUG1trUaNGqVQKKTi4mJNmzZNBw8eTDlm5syZCgQCKWPMmDEZbRoAkB+8QqixsVFz585VU1OT6uvrde7cOVVUVOjkyZMpx02ePFlHjhxJji1btmS0aQBAfujnc/DWrVtTXq9atUrFxcXavXu3xo8fn9weDAYViUQy0yEAIG9d1jOheDwuSSoqKkrZ3tDQoOLiYg0dOlSPPfaY2traLvpndHR0KJFIpAwAQN8QcM65dAqdc5o6daqOHz+uHTt2JLdv2LBB1157rcrKytTS0qKnnnpK586d0+7duxUMBjv9OTU1NXr66afT/xsAAHqleDyuwsLC7g9yaZozZ44rKytzra2t3R53+PBhV1BQ4DZu3Njl/jNnzrh4PJ4cra2tThKDwWAwcnzE4/HPzRKvZ0IXzJ8/X5s2bdL27ds1aNCgbo+NRqMqKytTc3Nzl/uDwWCXMyQAQP7zCiHnnObPn69XXnlFDQ0NKi8v/9yaY8eOqbW1VdFoNO0mAQD5yeuNCXPnztUf/vAHrVu3TqFQSLFYTLFYTKdPn5YknThxQk888YT+9re/6YMPPlBDQ4OmTJmigQMH6v7778/KXwAAkMN8ngPpIr/3W7VqlXPOuVOnTrmKigp3/fXXu4KCAnfjjTe6qqoqd+jQoUs+RzweN/89JoPBYDAuf1zKM6G03x2XLYlEQuFw2LoNAMBlupR3x7F2HADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwAwhBAAwQwgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADATK8LIeecdQsAgAy4lJ/nvS6E2tvbrVsAAGTApfw8D7heNvX49NNPdfjwYYVCIQUCgZR9iURCgwcPVmtrqwoLC406tMd1OI/rcB7X4Tyuw3m94To459Te3q7S0lJdcUX3c51+PdTTJbviiis0aNCgbo8pLCzs0zfZBVyH87gO53EdzuM6nGd9HcLh8CUd1+t+HQcA6DsIIQCAmZwKoWAwqCVLligYDFq3YorrcB7X4Tyuw3lch/Ny7Tr0ujcmAAD6jpyaCQEA8gshBAAwQwgBAMwQQgAAMzkVQi+88ILKy8t19dVXa+TIkdqxY4d1Sz2qpqZGgUAgZUQiEeu2sm779u2aMmWKSktLFQgE9Oqrr6bsd86ppqZGpaWl6t+/vyZMmKD9+/fbNJtFn3cdZs6c2en+GDNmjE2zWVJbW6tRo0YpFAqpuLhY06ZN08GDB1OO6Qv3w6Vch1y5H3ImhDZs2KAFCxZo8eLF2rNnj+666y5VVlbq0KFD1q31qFtvvVVHjhxJjn379lm3lHUnT57UiBEjVFdX1+X+ZcuWafny5aqrq9OuXbsUiUQ0adKkvFuH8POugyRNnjw55f7YsmVLD3aYfY2NjZo7d66amppUX1+vc+fOqaKiQidPnkwe0xfuh0u5DlKO3A8uR9xxxx1u1qxZKdtuueUW98Mf/tCoo563ZMkSN2LECOs2TElyr7zySvL1p59+6iKRiHvmmWeS286cOePC4bD71a9+ZdBhz/jsdXDOuaqqKjd16lSTfqy0tbU5Sa6xsdE513fvh89eB+dy537IiZnQ2bNntXv3blVUVKRsr6io0M6dO426stHc3KzS0lKVl5frwQcf1Pvvv2/dkqmWlhbFYrGUeyMYDOruu+/uc/eGJDU0NKi4uFhDhw7VY489pra2NuuWsioej0uSioqKJPXd++Gz1+GCXLgfciKEjh49qk8++UQlJSUp20tKShSLxYy66nmjR4/WmjVr9Nprr2nFihWKxWIaN26cjh07Zt2amQv///v6vSFJlZWVWrt2rbZt26Znn31Wu3bt0r333quOjg7r1rLCOafq6mrdeeedGjZsmKS+eT90dR2k3Lkfet0q2t357Fc7OOc6bctnlZWVyf8ePny4xo4dq5tuukmrV69WdXW1YWf2+vq9IUkzZsxI/vewYcN0++23q6ysTJs3b9b06dMNO8uOefPmae/evfrrX//aaV9fuh8udh1y5X7IiZnQwIEDdeWVV3b6l0xbW1unf/H0JQMGDNDw4cPV3Nxs3YqZC+8O5N7oLBqNqqysLC/vj/nz52vTpk164403Ur76pa/dDxe7Dl3prfdDToTQVVddpZEjR6q+vj5le319vcaNG2fUlb2Ojg4dOHBA0WjUuhUz5eXlikQiKffG2bNn1djY2KfvDUk6duyYWltb8+r+cM5p3rx5evnll7Vt2zaVl5en7O8r98PnXYeu9Nr7wfBNEV5eeuklV1BQ4H7729+6d955xy1YsMANGDDAffDBB9at9ZiFCxe6hoYG9/7777umpiZ33333uVAolPfXoL293e3Zs8ft2bPHSXLLly93e/bscR9++KFzzrlnnnnGhcNh9/LLL7t9+/a5hx56yEWjUZdIJIw7z6zurkN7e7tbuHCh27lzp2tpaXFvvPGGGzt2rLvhhhvy6jrMnj3bhcNh19DQ4I4cOZIcp06dSh7TF+6Hz7sOuXQ/5EwIOefc888/78rKytxVV13lbrvttpS3I/YFM2bMcNFo1BUUFLjS0lI3ffp0t3//fuu2su6NN95wkjqNqqoq59z5t+UuWbLERSIRFwwG3fjx492+fftsm86C7q7DqVOnXEVFhbv++utdQUGBu/HGG11VVZU7dOiQddsZ1dXfX5JbtWpV8pi+cD983nXIpfuBr3IAAJjJiWdCAID8RAgBAMwQQgAAM4QQAMAMIQQAMEMIAQDMEEIAADOEEADADCEEADBDCAEAzBBCAAAzhBAAwMz/AzhEJZ68xN5ZAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAYZ0lEQVR4nO3df2jU9x3H8ddV48W6y0HQ5O5mzELRbTQiVJ2aWX8UPDxYqLqBbWHEf6SdP0DSIrMyzDYwRajrH1kdK8Mpq9M/llpBaZuhiW7OLQ12tbaTOOO8TY/M4O5i1BPrZ3+IR89o9OJd3rm75wO+4H3v+/H78dsvefabu/uexznnBACAgSesJwAAKF5ECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmBlrPYF73b59WxcvXpTP55PH47GeDgAgQ8459ff3KxQK6Yknhr7WGXURunjxoqqqqqynAQB4TNFoVJMnTx5ym1H36zifz2c9BQBAFjzKz/OcRejtt99WTU2NSktLNXPmTB07duyRxvErOAAoDI/y8zwnEdq3b582bNigzZs36+TJk3r22WcViUR04cKFXOwOAJCnPLm4i/acOXP0zDPPaMeOHal13/72t7Vs2TI1NzcPOTaRSMjv92d7SgCAERaPx1VWVjbkNlm/Erp586a6uroUDofT1ofDYR0/fnzQ9slkUolEIm0BABSHrEfo8uXL+vLLL1VZWZm2vrKyUrFYbND2zc3N8vv9qYV3xgFA8cjZGxPufUHKOXffF6k2bdqkeDyeWqLRaK6mBAAYZbL+OaGJEydqzJgxg656ent7B10dSZLX65XX6832NAAAeSDrV0Ljxo3TzJkz1dbWlra+ra1NdXV12d4dACCP5eSOCY2NjfrhD3+oWbNmad68efr1r3+tCxcu6JVXXsnF7gAAeSonEVq5cqX6+vr0s5/9TJcuXVJtba0OHTqk6urqXOwOAJCncvI5ocfB54QAoDCYfE4IAIBHRYQAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMyMtZ4AgNHn2LFjGY9pbW3NeMwvfvGLjMegsHAlBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCY4QamQAGbNGnSsMZNmzYt4zF//vOfh7UvFDeuhAAAZogQAMBM1iPU1NQkj8eTtgQCgWzvBgBQAHLymtDTTz+tP/7xj6nHY8aMycVuAAB5LicRGjt2LFc/AICHyslrQt3d3QqFQqqpqdELL7ygc+fOPXDbZDKpRCKRtgAAikPWIzRnzhzt3r1bH374od555x3FYjHV1dWpr6/vvts3NzfL7/enlqqqqmxPCQAwSnmccy6XOxgYGNBTTz2ljRs3qrGxcdDzyWRSyWQy9TiRSBAiIEuG+zmhzz77LOMx9fX1GY/529/+lvEY5I94PK6ysrIht8n5h1UnTJig6dOnq7u7+77Pe71eeb3eXE8DADAK5fxzQslkUl988YWCwWCudwUAyDNZj9Brr72mjo4O9fT06K9//at+8IMfKJFIqKGhIdu7AgDkuaz/Ou7f//63XnzxRV2+fFmTJk3S3LlzdeLECVVXV2d7VwCAPJf1CO3duzfbfyUADe9D362trcPa19ixmf9o+Oc//zmsfaG4ce84AIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMBMzr/UDkB2DOebS+fPnz+sfe3bty/jMX19fcPaF4obV0IAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAww120gTwxc+bMEdtXV1fXiO0LxY0rIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADDcwBfJEaWnpiO3r7NmzI7YvFDeuhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAMx7nnLOexFclEgn5/X7raQA5VVJSkvGYzz//POMxV69ezXiMJM2bNy/jMTdu3BjWvlC44vG4ysrKhtyGKyEAgBkiBAAwk3GEjh49qvr6eoVCIXk8Hu3fvz/teeecmpqaFAqFNH78eC1atEinT5/O1nwBAAUk4wgNDAxoxowZamlpue/z27Zt0/bt29XS0qLOzk4FAgEtWbJE/f39jz1ZAEBhyfibVSORiCKRyH2fc87prbfe0ubNm7VixQpJ0q5du1RZWak9e/bo5ZdffrzZAgAKSlZfE+rp6VEsFlM4HE6t83q9WrhwoY4fP37fMclkUolEIm0BABSHrEYoFotJkiorK9PWV1ZWpp67V3Nzs/x+f2qpqqrK5pQAAKNYTt4d5/F40h475watu2vTpk2Kx+OpJRqN5mJKAIBRKOPXhIYSCAQk3bkiCgaDqfW9vb2Dro7u8nq98nq92ZwGACBPZPVKqKamRoFAQG1tbal1N2/eVEdHh+rq6rK5KwBAAcj4Sujq1as6e/Zs6nFPT48++eQTlZeXa8qUKdqwYYO2bt2qqVOnaurUqdq6dauefPJJvfTSS1mdOAAg/2UcoY8//liLFy9OPW5sbJQkNTQ06Le//a02btyo69eva82aNbpy5YrmzJmjjz76SD6fL3uzBgAUBG5gChhYvnx5xmNaW1szHvP6669nPEa6865V4HFxA1MAwKhGhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM1n9ZlUAj2akvuTxH//4x4jsBxguroQAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADMe55yznsRXJRIJ+f1+62kAj6y2tjbjMZ2dnRmPOX36dMZjvvvd72Y8RpKSyeSwxgFfFY/HVVZWNuQ2XAkBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGbGWk8AyHf19fUZjyktLc14zPvvv5/xGG5EitGOKyEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAw3MAUe0/Tp00dkP5999tmI7AcYSVwJAQDMECEAgJmMI3T06FHV19crFArJ4/Fo//79ac+vWrVKHo8nbZk7d2625gsAKCAZR2hgYEAzZsxQS0vLA7dZunSpLl26lFoOHTr0WJMEABSmjN+YEIlEFIlEhtzG6/UqEAgMe1IAgOKQk9eE2tvbVVFRoWnTpmn16tXq7e194LbJZFKJRCJtAQAUh6xHKBKJ6N1339Xhw4f15ptvqrOzU88999wDv+u+ublZfr8/tVRVVWV7SgCAUSrrnxNauXJl6s+1tbWaNWuWqqurdfDgQa1YsWLQ9ps2bVJjY2PqcSKRIEQAUCRy/mHVYDCo6upqdXd33/d5r9crr9eb62kAAEahnH9OqK+vT9FoVMFgMNe7AgDkmYyvhK5evaqzZ8+mHvf09OiTTz5ReXm5ysvL1dTUpO9///sKBoM6f/68Xn/9dU2cOFHLly/P6sQBAPkv4wh9/PHHWrx4cerx3ddzGhoatGPHDp06dUq7d+/W//73PwWDQS1evFj79u2Tz+fL3qwBAAXB45xz1pP4qkQiIb/fbz0NFKnS0tKMx/z973/PeMzt27czHjN//vyMx/T19WU8BsiWeDyusrKyIbfh3nEAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwk/NvVgXySSQSyXjMtGnTMh7z85//POMx3BEbhYgrIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADDcwBb6irq5uRPbz3//+d0T2A4x2XAkBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGa4gSkKUmlp6bDG1dfXZzzm+vXrGY85ePBgxmOAQsSVEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghhuYoiBFIpFhjfvmN7+Z8Zi9e/dmPObcuXMZjwEKEVdCAAAzRAgAYCajCDU3N2v27Nny+XyqqKjQsmXLdObMmbRtnHNqampSKBTS+PHjtWjRIp0+fTqrkwYAFIaMItTR0aG1a9fqxIkTamtr061btxQOhzUwMJDaZtu2bdq+fbtaWlrU2dmpQCCgJUuWqL+/P+uTBwDkt4zemPDBBx+kPd65c6cqKirU1dWlBQsWyDmnt956S5s3b9aKFSskSbt27VJlZaX27Nmjl19+OXszBwDkvcd6TSgej0uSysvLJUk9PT2KxWIKh8OpbbxerxYuXKjjx4/f9+9IJpNKJBJpCwCgOAw7Qs45NTY2av78+aqtrZUkxWIxSVJlZWXatpWVlann7tXc3Cy/359aqqqqhjslAECeGXaE1q1bp08//VS///3vBz3n8XjSHjvnBq27a9OmTYrH46klGo0Od0oAgDwzrA+rrl+/XgcOHNDRo0c1efLk1PpAICDpzhVRMBhMre/t7R10dXSX1+uV1+sdzjQAAHkuoysh55zWrVun1tZWHT58WDU1NWnP19TUKBAIqK2tLbXu5s2b6ujoUF1dXXZmDAAoGBldCa1du1Z79uzR+++/L5/Pl3qdx+/3a/z48fJ4PNqwYYO2bt2qqVOnaurUqdq6dauefPJJvfTSSzn5BwAA8ldGEdqxY4ckadGiRWnrd+7cqVWrVkmSNm7cqOvXr2vNmjW6cuWK5syZo48++kg+ny8rEwYAFI6MIuSce+g2Ho9HTU1NampqGu6cgMf2jW98Y8T21dPTM2L7AgoN944DAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM2OtJwDkwn/+858R21c0Gh2xfQGFhishAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMCMxznnrCfxVYlEQn6/33oaAIDHFI/HVVZWNuQ2XAkBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAMxlFqLm5WbNnz5bP51NFRYWWLVumM2fOpG2zatUqeTyetGXu3LlZnTQAoDBkFKGOjg6tXbtWJ06cUFtbm27duqVwOKyBgYG07ZYuXapLly6llkOHDmV10gCAwjA2k40/+OCDtMc7d+5URUWFurq6tGDBgtR6r9erQCCQnRkCAArWY70mFI/HJUnl5eVp69vb21VRUaFp06Zp9erV6u3tfeDfkUwmlUgk0hYAQHHwOOfccAY65/T888/rypUrOnbsWGr9vn379LWvfU3V1dXq6enRT37yE926dUtdXV3yer2D/p6mpib99Kc/Hf6/AAAwKsXjcZWVlQ29kRumNWvWuOrqaheNRofc7uLFi66kpMT94Q9/uO/zN27ccPF4PLVEo1EniYWFhYUlz5d4PP7QlmT0mtBd69ev14EDB3T06FFNnjx5yG2DwaCqq6vV3d193+e9Xu99r5AAAIUvowg557R+/Xq99957am9vV01NzUPH9PX1KRqNKhgMDnuSAIDClNEbE9auXavf/e532rNnj3w+n2KxmGKxmK5fvy5Junr1ql577TX95S9/0fnz59Xe3q76+npNnDhRy5cvz8k/AACQxzJ5HUgP+L3fzp07nXPOXbt2zYXDYTdp0iRXUlLipkyZ4hoaGtyFCxceeR/xeNz895gsLCwsLI+/PMprQsN+d1yuJBIJ+f1+62kAAB7To7w7jnvHAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMjLoIOeespwAAyIJH+Xk+6iLU399vPQUAQBY8ys9zjxtllx63b9/WxYsX5fP55PF40p5LJBKqqqpSNBpVWVmZ0QztcRzu4DjcwXG4g+Nwx2g4Ds459ff3KxQK6Yknhr7WGTtCc3pkTzzxhCZPnjzkNmVlZUV9kt3FcbiD43AHx+EOjsMd1sfB7/c/0naj7tdxAIDiQYQAAGbyKkJer1dbtmyR1+u1noopjsMdHIc7OA53cBzuyLfjMOremAAAKB55dSUEACgsRAgAYIYIAQDMECEAgJm8itDbb7+tmpoalZaWaubMmTp27Jj1lEZUU1OTPB5P2hIIBKynlXNHjx5VfX29QqGQPB6P9u/fn/a8c05NTU0KhUIaP368Fi1apNOnT9tMNocedhxWrVo16PyYO3euzWRzpLm5WbNnz5bP51NFRYWWLVumM2fOpG1TDOfDoxyHfDkf8iZC+/bt04YNG7R582adPHlSzz77rCKRiC5cuGA9tRH19NNP69KlS6nl1KlT1lPKuYGBAc2YMUMtLS33fX7btm3avn27Wlpa1NnZqUAgoCVLlhTcfQgfdhwkaenSpWnnx6FDh0ZwhrnX0dGhtWvX6sSJE2pra9OtW7cUDoc1MDCQ2qYYzodHOQ5SnpwPLk985zvfca+88kraum9961vuxz/+sdGMRt6WLVvcjBkzrKdhSpJ77733Uo9v377tAoGAe+ONN1Lrbty44fx+v/vVr35lMMORce9xcM65hoYG9/zzz5vMx0pvb6+T5Do6OpxzxXs+3HscnMuf8yEvroRu3ryprq4uhcPhtPXhcFjHjx83mpWN7u5uhUIh1dTU6IUXXtC5c+esp2Sqp6dHsVgs7dzwer1auHBh0Z0bktTe3q6KigpNmzZNq1evVm9vr/WUcioej0uSysvLJRXv+XDvcbgrH86HvIjQ5cuX9eWXX6qysjJtfWVlpWKxmNGsRt6cOXO0e/duffjhh3rnnXcUi8VUV1envr4+66mZufvfv9jPDUmKRCJ69913dfjwYb355pvq7OzUc889p2QyaT21nHDOqbGxUfPnz1dtba2k4jwf7nccpPw5H0bdXbSHcu9XOzjnBq0rZJFIJPXn6dOna968eXrqqae0a9cuNTY2Gs7MXrGfG5K0cuXK1J9ra2s1a9YsVVdX6+DBg1qxYoXhzHJj3bp1+vTTT/WnP/1p0HPFdD486Djky/mQF1dCEydO1JgxYwb9n0xvb++g/+MpJhMmTND06dPV3d1tPRUzd98dyLkxWDAYVHV1dUGeH+vXr9eBAwd05MiRtK9+Kbbz4UHH4X5G6/mQFxEaN26cZs6cqba2trT1bW1tqqurM5qVvWQyqS+++ELBYNB6KmZqamoUCATSzo2bN2+qo6OjqM8NSerr61M0Gi2o88M5p3Xr1qm1tVWHDx9WTU1N2vPFcj487Djcz6g9HwzfFJGRvXv3upKSEveb3/zGff75527Dhg1uwoQJ7vz589ZTGzGvvvqqa29vd+fOnXMnTpxw3/ve95zP5yv4Y9Df3+9OnjzpTp486SS57du3u5MnT7p//etfzjnn3njjDef3+11ra6s7deqUe/HFF10wGHSJRMJ45tk11HHo7+93r776qjt+/Ljr6elxR44ccfPmzXNf//rXC+o4/OhHP3J+v9+1t7e7S5cupZZr166ltimG8+FhxyGfzoe8iZBzzv3yl7901dXVbty4ce6ZZ55JeztiMVi5cqULBoOupKTEhUIht2LFCnf69GnraeXckSNHnKRBS0NDg3Puzttyt2zZ4gKBgPN6vW7BggXu1KlTtpPOgaGOw7Vr11w4HHaTJk1yJSUlbsqUKa6hocFduHDBetpZdb9/vyS3c+fO1DbFcD487Djk0/nAVzkAAMzkxWtCAIDCRIQAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCY+T/t0XoLNJYjiQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAaAElEQVR4nO3df2jU9x3H8ddp9Wptcl3Q5C4zhuCUjeoEf1QNrcYOgxmVpm5g2zHiKKIz2oW0yJwMYwemE2oLy+q2UpxSbR2bdbJKbYYmsUsdKorWtZJinCmaZWb2LkZNFv3sD/HYmfjje97lnbs8H/AFc3dv7+O33/r0m7v7xueccwIAwMAQ6wUAAAYvIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMw8YL2AW12/fl3nzp1TRkaGfD6f9XIAAB4559TR0aHc3FwNGXLnc50BF6Fz584pLy/PehkAgPvU0tKiMWPG3PExA+7bcRkZGdZLAAAkwL38fZ60CL355psqKCjQgw8+qKlTp+rAgQP3NMe34AAgPdzL3+dJidCOHTtUUVGhNWvW6OjRo3riiSdUUlKis2fPJuPpAAApypeMq2jPmDFDU6ZM0aZNm6K3fetb31Jpaamqq6vvOBuJRBQIBBK9JABAPwuHw8rMzLzjYxJ+JtTd3a0jR46ouLg45vbi4mI1Njb2enxXV5cikUjMBgAYHBIeoQsXLujatWvKycmJuT0nJ0etra29Hl9dXa1AIBDdeGccAAweSXtjwq0vSDnn+nyRavXq1QqHw9GtpaUlWUsCAAwwCf+c0KhRozR06NBeZz1tbW29zo4kye/3y+/3J3oZAIAUkPAzoeHDh2vq1Kmqra2Nub22tlaFhYWJfjoAQApLyhUTKisr9cMf/lDTpk3TrFmz9Lvf/U5nz57VsmXLkvF0AIAUlZQILVq0SO3t7XrllVd0/vx5TZw4UXv27FF+fn4yng4AkKKS8jmh+8HnhAAgPZh8TggAgHtFhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJhJylW0AaS2kSNHep45duyY55kf/ehHnmc+/vhjzzMYuDgTAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBmuog2gl1Ao5Hnm2rVrSVgJ0h1nQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGS5gCqCX6upqzzMTJkxIwkqQ7jgTAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMcAFTII3NmzcvrrmioiLPM11dXZ5nvvrqK88zSC+cCQEAzBAhAICZhEeoqqpKPp8vZgsGg4l+GgBAGkjKa0KPPvqo/vrXv0a/Hjp0aDKeBgCQ4pISoQceeICzHwDAXSXlNaGmpibl5uaqoKBAzz77rE6fPn3bx3Z1dSkSicRsAIDBIeERmjFjhrZu3aq9e/fqrbfeUmtrqwoLC9Xe3t7n46urqxUIBKJbXl5eopcEABigfM45l8wn6Ozs1Lhx47Rq1SpVVlb2ur+rqyvm8wWRSIQQAQkS7+eEtm/f7nnm4Ycf9jwzffp0zzOffvqp5xnYCIfDyszMvONjkv5h1ZEjR2rSpElqamrq836/3y+/35/sZQAABqCkf06oq6tLn332mUKhULKfCgCQYhIeoZdffln19fVqbm7W3//+d33/+99XJBJRWVlZop8KAJDiEv7tuC+//FLPPfecLly4oNGjR2vmzJk6ePCg8vPzE/1UAIAUl/AIvffee4n+LYG0E8+L+Bs2bPA8s2TJEs8zUnwfMH/hhRc8z/AmA3DtOACAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADATNJ/qB2A3pYuXep5ZtmyZUlYSd/WrVvneeadd95JwkqQ7jgTAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBmuog3cp29/+9ueZ9asWZOElfT29ttvxzVXXV3teea///1vXM+FwY0zIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBcwBf7PyJEjPc/88Y9/9DzzyCOPeJ75wx/+4Hkm3guldnd3xzUHeMWZEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghguYIi3FcyFSSfrb3/7meeYb3/iG55lTp055nlm5cqXnmX//+9+eZ4D+xJkQAMAMEQIAmPEcoYaGBi1YsEC5ubny+XzatWtXzP3OOVVVVSk3N1cjRoxQUVGRTp48maj1AgDSiOcIdXZ2avLkyaqpqenz/g0bNmjjxo2qqanRoUOHFAwGNW/ePHV0dNz3YgEA6cXzGxNKSkpUUlLS533OOb3xxhtas2aNFi5cKEnasmWLcnJytH37di1duvT+VgsASCsJfU2oublZra2tKi4ujt7m9/s1Z84cNTY29jnT1dWlSCQSswEABoeERqi1tVWSlJOTE3N7Tk5O9L5bVVdXKxAIRLe8vLxELgkAMIAl5d1xPp8v5mvnXK/bblq9erXC4XB0a2lpScaSAAADUEI/rBoMBiXdOCMKhULR29va2nqdHd3k9/vl9/sTuQwAQIpI6JlQQUGBgsGgamtro7d1d3ervr5ehYWFiXwqAEAa8HwmdOnSJX3xxRfRr5ubm3Xs2DFlZWVp7Nixqqio0Pr16zV+/HiNHz9e69ev10MPPaTnn38+oQsHAKQ+zxE6fPiw5s6dG/26srJSklRWVqbf//73WrVqla5cuaLly5fr4sWLmjFjhj766CNlZGQkbtUAgLTgc84560X8v0gkokAgYL0MpLidO3fGNVdaWup55urVq55nHnvsMc8zn376qecZwFI4HFZmZuYdH8O14wAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGAmoT9ZFUiG8vJyzzNPPfVUElbSt5KSEs8zra2tnmcqKio8z3zyySeeZyRpxYoVnme+/PJLzzO/+MUvPM9cvnzZ8wwGLs6EAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzXMAU/WrEiBGeZ1555RXPMw88EN+hvWXLFs8zp0+f9jwTz4VFx40b53kmnouKStILL7zgeWb37t2eZ9ra2jzPvP76655nMHBxJgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmOECpuhX5eXlnme+9rWveZ7p7u72PCNJ69at8zwTz0VP++tipCUlJZ5nJOnkyZOeZxobGz3P/PKXv/Q8c+DAAc8zhw8f9jyD/sGZEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABgxuecc9aL+H+RSESBQMB6GUiS//znP55nHnnkEc8zv/rVrzzPSNKkSZM8zxQVFXme6enp6ZfnieeiovGaP3++55ldu3Z5nvnLX/7ieeYHP/iB5xlJ6urqimsON4TDYWVmZt7xMZwJAQDMECEAgBnPEWpoaNCCBQuUm5srn8/X63R68eLF8vl8MdvMmTMTtV4AQBrxHKHOzk5NnjxZNTU1t33M/Pnzdf78+ei2Z8+e+1okACA9ef7JqiUlJXf9aY1+v1/BYDDuRQEABoekvCZUV1en7OxsTZgwQUuWLFFbW9ttH9vV1aVIJBKzAQAGh4RHqKSkRNu2bdO+ffv02muv6dChQ3ryySdv+1bH6upqBQKB6JaXl5foJQEABijP3467m0WLFkV/PXHiRE2bNk35+fn64IMPtHDhwl6PX716tSorK6NfRyIRQgQAg0TCI3SrUCik/Px8NTU19Xm/3++X3+9P9jIAAANQ0j8n1N7erpaWFoVCoWQ/FQAgxXg+E7p06ZK++OKL6NfNzc06duyYsrKylJWVpaqqKn3ve99TKBTSmTNn9LOf/UyjRo3SM888k9CFAwBSn+cIHT58WHPnzo1+ffP1nLKyMm3atEknTpzQ1q1b9dVXXykUCmnu3LnasWOHMjIyErdqAEBa4AKmiNvw4cM9z/zrX//yPDPQj4d4Lkb6k5/8xPPMpk2bPM8MdLW1tZ5nvvOd73ieKS8v9zwjpec+709cwBQAMKARIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADATNJ/sirSV1FRkeeZgX5F7GvXrnme4YrYNwwZ4v3ftJ9//rnnmXiuoj1lyhTPM+gfnAkBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGa4gCniNmHCBOsl3Nb169fjmnvxxRc9z6TjxUhHjx7teea3v/2t55nS0lLPM/HYt29fvzwPvONMCABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwwwVMEbf169dbL+G2Ghoa4pr7/PPPPc/Ec7HP/Px8zzMTJ070PFNcXOx5RpK++93vep7JzMyM67m8ev311z3PvPvuu0lYCRKBMyEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwIzPOeesF/H/IpGIAoGA9TJwD44fP+55Jp6LcPYnn8/neWaA/S8UI54/jyT19PR4nqmtrfU8s23btn6ZgY1wOHzXC9tyJgQAMEOEAABmPEWourpa06dPV0ZGhrKzs1VaWqpTp07FPMY5p6qqKuXm5mrEiBEqKirSyZMnE7poAEB68BSh+vp6lZeX6+DBg6qtrVVPT4+Ki4vV2dkZfcyGDRu0ceNG1dTU6NChQwoGg5o3b546OjoSvngAQGrz9JNVP/zww5ivN2/erOzsbB05ckSzZ8+Wc05vvPGG1qxZo4ULF0qStmzZopycHG3fvl1Lly5N3MoBACnvvl4TCofDkqSsrCxJUnNzs1pbW2N+pLDf79ecOXPU2NjY5+/R1dWlSCQSswEABoe4I+ScU2VlpR5//PHo225bW1slSTk5OTGPzcnJid53q+rqagUCgeiWl5cX75IAACkm7gitWLFCx48f17vvvtvrvls/m+Ccu+3nFVavXq1wOBzdWlpa4l0SACDFeHpN6KaVK1dq9+7damho0JgxY6K3B4NBSTfOiEKhUPT2tra2XmdHN/n9fvn9/niWAQBIcZ7OhJxzWrFihXbu3Kl9+/apoKAg5v6CggIFg8GYT053d3ervr5ehYWFiVkxACBteDoTKi8v1/bt2/XnP/9ZGRkZ0dd5AoGARowYIZ/Pp4qKCq1fv17jx4/X+PHjtX79ej300EN6/vnnk/IHAACkLk8R2rRpkySpqKgo5vbNmzdr8eLFkqRVq1bpypUrWr58uS5evKgZM2boo48+UkZGRkIWDABIH1zAFHG73et8d/Liiy96nqmsrPQ8M9BfZ4znowi3fk7vXuzdu9fzjCTt2rXL88zFixfjei6kLy5gCgAY0IgQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGq2gDAJKCq2gDAAY0IgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBlPEaqurtb06dOVkZGh7OxslZaW6tSpUzGPWbx4sXw+X8w2c+bMhC4aAJAePEWovr5e5eXlOnjwoGpra9XT06Pi4mJ1dnbGPG7+/Pk6f/58dNuzZ09CFw0ASA8PeHnwhx9+GPP15s2blZ2drSNHjmj27NnR2/1+v4LBYGJWCABIW/f1mlA4HJYkZWVlxdxeV1en7OxsTZgwQUuWLFFbW9ttf4+uri5FIpGYDQAwOPiccy6eQeecnn76aV28eFEHDhyI3r5jxw49/PDDys/PV3Nzs37+85+rp6dHR44ckd/v7/X7VFVVad26dfH/CQAAA1I4HFZmZuadH+TitHz5cpefn+9aWlru+Lhz5865YcOGuT/96U993n/16lUXDoejW0tLi5PExsbGxpbiWzgcvmtLPL0mdNPKlSu1e/duNTQ0aMyYMXd8bCgUUn5+vpqamvq83+/393mGBABIf54i5JzTypUr9f7776uurk4FBQV3nWlvb1dLS4tCoVDciwQApCdPb0woLy/XO++8o+3btysjI0Otra1qbW3VlStXJEmXLl3Syy+/rE8++URnzpxRXV2dFixYoFGjRumZZ55Jyh8AAJDCvLwOpNt832/z5s3OOecuX77siouL3ejRo92wYcPc2LFjXVlZmTt79uw9P0c4HDb/PiYbGxsb2/1v9/KaUNzvjkuWSCSiQCBgvQwAwH26l3fHce04AIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAICZARch55z1EgAACXAvf58PuAh1dHRYLwEAkAD38ve5zw2wU4/r16/r3LlzysjIkM/ni7kvEokoLy9PLS0tyszMNFqhPfbDDeyHG9gPN7AfbhgI+8E5p46ODuXm5mrIkDuf6zzQT2u6Z0OGDNGYMWPu+JjMzMxBfZDdxH64gf1wA/vhBvbDDdb7IRAI3NPjBty34wAAgwcRAgCYSakI+f1+rV27Vn6/33opptgPN7AfbmA/3MB+uCHV9sOAe2MCAGDwSKkzIQBAeiFCAAAzRAgAYIYIAQDMpFSE3nzzTRUUFOjBBx/U1KlTdeDAAesl9auqqir5fL6YLRgMWi8r6RoaGrRgwQLl5ubK5/Np165dMfc751RVVaXc3FyNGDFCRUVFOnnypM1ik+hu+2Hx4sW9jo+ZM2faLDZJqqurNX36dGVkZCg7O1ulpaU6depUzGMGw/FwL/shVY6HlInQjh07VFFRoTVr1ujo0aN64oknVFJSorNnz1ovrV89+uijOn/+fHQ7ceKE9ZKSrrOzU5MnT1ZNTU2f92/YsEEbN25UTU2NDh06pGAwqHnz5qXddQjvth8kaf78+THHx549e/pxhclXX1+v8vJyHTx4ULW1terp6VFxcbE6OzujjxkMx8O97AcpRY4HlyIee+wxt2zZspjbvvnNb7qf/vSnRivqf2vXrnWTJ0+2XoYpSe7999+Pfn39+nUXDAbdq6++Gr3t6tWrLhAIuN/85jcGK+wft+4H55wrKytzTz/9tMl6rLS1tTlJrr6+3jk3eI+HW/eDc6lzPKTEmVB3d7eOHDmi4uLimNuLi4vV2NhotCobTU1Nys3NVUFBgZ599lmdPn3aekmmmpub1draGnNs+P1+zZkzZ9AdG5JUV1en7OxsTZgwQUuWLFFbW5v1kpIqHA5LkrKysiQN3uPh1v1wUyocDykRoQsXLujatWvKycmJuT0nJ0etra1Gq+p/M2bM0NatW7V371699dZbam1tVWFhodrb262XZubmf//BfmxIUklJibZt26Z9+/bptdde06FDh/Tkk0+qq6vLemlJ4ZxTZWWlHn/8cU2cOFHS4Dwe+toPUuocDwPuKtp3cuuPdnDO9botnZWUlER/PWnSJM2aNUvjxo3Tli1bVFlZabgye4P92JCkRYsWRX89ceJETZs2Tfn5+frggw+0cOFCw5Ulx4oVK3T8+HF9/PHHve4bTMfD7fZDqhwPKXEmNGrUKA0dOrTXv2Ta2tp6/YtnMBk5cqQmTZqkpqYm66WYufnuQI6N3kKhkPLz89Py+Fi5cqV2796t/fv3x/zol8F2PNxuP/RloB4PKRGh4cOHa+rUqaqtrY25vba2VoWFhUarstfV1aXPPvtMoVDIeilmCgoKFAwGY46N7u5u1dfXD+pjQ5La29vV0tKSVseHc04rVqzQzp07tW/fPhUUFMTcP1iOh7vth74M2OPB8E0Rnrz33ntu2LBh7u2333b/+Mc/XEVFhRs5cqQ7c+aM9dL6zUsvveTq6urc6dOn3cGDB91TTz3lMjIy0n4fdHR0uKNHj7qjR486SW7jxo3u6NGj7p///KdzzrlXX33VBQIBt3PnTnfixAn33HPPuVAo5CKRiPHKE+tO+6Gjo8O99NJLrrGx0TU3N7v9+/e7WbNmua9//etptR9+/OMfu0Ag4Orq6tz58+ej2+XLl6OPGQzHw932QyodDykTIeec+/Wvf+3y8/Pd8OHD3ZQpU2LejjgYLFq0yIVCITds2DCXm5vrFi5c6E6ePGm9rKTbv3+/k9RrKysrc87deFvu2rVrXTAYdH6/382ePdudOHHCdtFJcKf9cPnyZVdcXOxGjx7thg0b5saOHevKysrc2bNnrZedUH39+SW5zZs3Rx8zGI6Hu+2HVDoe+FEOAAAzKfGaEAAgPREhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZv4H69b9p0avm1EAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 请在此处完成代码并作图\n",
    "selected_numbers = [1,2,3,4,5]\n",
    "selected_images = x[selected_numbers].squeeze().cpu().numpy()\n",
    "selected_labels = y[selected_numbers]\n",
    "for i in range(5):\n",
    "    plt.imshow(selected_images[i],cmap=\"gray\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0281c1d2",
   "metadata": {},
   "source": [
    "### 2.2 定义模型结构"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35776058",
   "metadata": {},
   "source": [
    "我们搭建一个类似于 LeNet-5 的网络，结构如下："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d175cb17",
   "metadata": {},
   "source": [
    "![](https://pic1.zhimg.com/80/v2-82eabb4c17e90d467197d013f7629f3c_720w.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ec06760",
   "metadata": {},
   "source": [
    "我们需要创建2个卷积层、2个汇聚层和2个全连接层，**暂时忽略所有的激活函数**。所有隐藏层的函数细节都可以在[官方文档](https://pytorch.org/docs/stable/nn.html)中按分类找到。每一个隐藏层本质上都是将一个数组变换成另一个数组的函数，因此为了确认编写的模型是正确的，可以先用一个小数据进行测试，观察输入和输出的维度。例如，我们先取出前6个观测，此时输入的维度是 `[6, 1, 28, 28]`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a26ebf44",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 1, 28, 28])\n",
      "torch.Size([6])\n"
     ]
    }
   ],
   "source": [
    "ns = 6\n",
    "smallx = x[0:ns]\n",
    "smally = y[0:ns]\n",
    "print(smallx.shape)\n",
    "print(smally.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ada326cb",
   "metadata": {},
   "source": [
    "接下来创建第1个卷积层，并测试输出的维度。注意到我们可以直接将隐藏层当成一个函数来调用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "500c1874",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 20, 24, 24])\n"
     ]
    }
   ],
   "source": [
    "conv1 = torch.nn.Conv2d(in_channels=1, out_channels=20, kernel_size=5, stride=1)\n",
    "res1 = conv1(smallx)\n",
    "print(res1.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2e8230d",
   "metadata": {},
   "source": [
    "可以看到，输出的维度为 `[20, 24, 24]`（不包括第1位的数据批次维度），与之前图中的结果吻合。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfc3c891",
   "metadata": {},
   "source": [
    "接下来，请按照图中提示编写层对象 `pool1`、`conv2`、`pool2`、`fc1` 和 `fc2`，并顺次测试输入与输出的维度，使其与上图匹配。注意，在将一个大小为 `[6, 50, 4, 4]` 的数组（假设叫 `somearray`）传递给 `fc1` 之前，需要先将其变形为只有两个维度的数组，做法是 `somearray.view(-1, 50 * 4 * 4)`，其中 -1 表示该位置的大小不变。也可以使用 `torch.flatten()` 函数并指定其中的 `start_dim` 参数（请搜索其对应的函数文档）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94e3f58c",
   "metadata": {},
   "source": [
    "```py\n",
    "pool1 = ...\n",
    "res2 = pool1(res1)\n",
    "print(res2.shape)\n",
    "assert res2.shape == (ns, 20, 12, 12), \"pool1 输出形状不对\"\n",
    "\n",
    "conv2 = ...\n",
    "res3 = conv2(res2)\n",
    "print(res3.shape)\n",
    "assert res3.shape == (ns, 50, 8, 8), \"conv2 输出形状不对\"\n",
    "\n",
    "pool2 = ...\n",
    "res4 = pool2(res3)\n",
    "print(res4.shape)\n",
    "assert res4.shape == (ns, 50, 4, 4), \"pool2 输出形状不对\"\n",
    "\n",
    "fc1 = ...\n",
    "res5 = fc1(res4.view(-1, 800))\n",
    "print(res5.shape)\n",
    "assert res5.shape == (ns, 500), \"fc1 输出形状不对\"\n",
    "\n",
    "fc2 = ...\n",
    "res6 = fc2(res5)\n",
    "print(res6.shape)\n",
    "assert res6.shape == (ns, 10), \"fc2 输出形状不对\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89164f79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 请在此处完成上述代码\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "45bf6da6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 20, 12, 12])\n"
     ]
    }
   ],
   "source": [
    "pool1 = torch.nn.MaxPool2d(kernel_size=2)\n",
    "res2 = pool1(res1)\n",
    "print(res2.shape)\n",
    "assert res2.shape == (ns, 20, 12, 12), \"pool1 输出形状不对\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "51c066dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 50, 8, 8])\n"
     ]
    }
   ],
   "source": [
    "conv2 = torch.nn.Conv2d(in_channels=20, out_channels=50, kernel_size=5, stride=1)\n",
    "res3 = conv2(res2)\n",
    "print(res3.shape)\n",
    "assert res3.shape == (ns, 50, 8, 8), \"conv2 输出形状不对\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0d9a034c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 50, 4, 4])\n"
     ]
    }
   ],
   "source": [
    "pool2 = torch.nn.MaxPool2d(kernel_size=2)\n",
    "res4 = pool2(res3)\n",
    "print(res4.shape)\n",
    "assert res4.shape == (ns, 50, 4, 4), \"pool2 输出形状不对\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a33f8f6f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 500])\n"
     ]
    }
   ],
   "source": [
    "fc1 = torch.nn.Linear(in_features=800, out_features=500)\n",
    "res5 = fc1(res4.view(-1, 800))\n",
    "print(res5.shape)\n",
    "assert res5.shape == (ns, 500), \"fc1 输出形状不对\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8e6c61e0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 10])\n"
     ]
    }
   ],
   "source": [
    "fc2 = torch.nn.Linear(in_features=500, out_features=10)\n",
    "res6 = fc2(res5)\n",
    "print(res6.shape)\n",
    "assert res6.shape == (ns, 10), \"fc2 输出形状不对\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f623e96f",
   "metadata": {},
   "source": [
    "### 2.3 创建模型类"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91ba53a4",
   "metadata": {},
   "source": [
    "在确保隐藏层维度都正确后，将所有的隐藏层封装到一个模型类中，其中模型结构在 `__init__()` 中定义，具体的计算过程在 `forward()` 中实现。此时需要加入激活函数。在本模型中，**请在 `conv1`、`conv2` 和 `fc1` 后加入 ReLU 激活函数，并在 `fc2` 后加入 Softmax 激活函数**。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "415791e4",
   "metadata": {},
   "source": [
    "```py\n",
    "class MyModel(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = ...\n",
    "        self.pool1 = ...\n",
    "        self.conv2 = ...\n",
    "        self.pool2 = ...\n",
    "        self.fc1 = ...\n",
    "        self.fc2 = ...\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.pool1(x)\n",
    "        ...\n",
    "        return x\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9f6b79df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 请在此处完成上述代码\n",
    "class MyModel(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = torch.nn.Conv2d(in_channels=1, out_channels=20, kernel_size=5, stride=1)\n",
    "        self.pool1 = torch.nn.MaxPool2d(kernel_size=2)\n",
    "        self.conv2 = torch.nn.Conv2d(in_channels=20, out_channels=50, kernel_size=5, stride=1)\n",
    "        self.pool2 = torch.nn.MaxPool2d(kernel_size=2)\n",
    "        self.fc1 = torch.nn.Linear(in_features=800, out_features=500)\n",
    "        self.fc2 = torch.nn.Linear(in_features=500, out_features=10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.pool1(x)\n",
    "        x = self.conv2(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.pool2(x)\n",
    "        x = torch.flatten(x, start_dim=1)\n",
    "        x = self.fc1(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        x = torch.nn.functional.softmax(x, dim=1)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49894cba",
   "metadata": {},
   "source": [
    "再次测试输入输出的维度是否正确。如果模型编写正确，输出的维度应该是 `[6, 10]`，且输出结果为0到1之间的概率值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "95558fc5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 10])\n",
      "\n",
      "tensor([[0.0975, 0.1003, 0.0987, 0.1080, 0.0927, 0.0913, 0.0983, 0.1060, 0.1013,\n",
      "         0.1058],\n",
      "        [0.0971, 0.1000, 0.1006, 0.1078, 0.0939, 0.0914, 0.1014, 0.1017, 0.1025,\n",
      "         0.1037],\n",
      "        [0.0958, 0.0996, 0.0984, 0.1092, 0.0932, 0.0923, 0.0984, 0.1066, 0.1022,\n",
      "         0.1043],\n",
      "        [0.0991, 0.1006, 0.0990, 0.1091, 0.0918, 0.0889, 0.0989, 0.1061, 0.1042,\n",
      "         0.1021],\n",
      "        [0.0954, 0.1004, 0.1015, 0.1071, 0.0943, 0.0922, 0.1017, 0.1023, 0.1027,\n",
      "         0.1024],\n",
      "        [0.0962, 0.1011, 0.0987, 0.1094, 0.0923, 0.0906, 0.0958, 0.1088, 0.1024,\n",
      "         0.1048]], grad_fn=<SoftmaxBackward0>)\n",
      "\n",
      "tensor([1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],\n",
      "       grad_fn=<SumBackward1>)\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(123)\n",
    "torch.manual_seed(123)\n",
    "\n",
    "model = MyModel()\n",
    "pred = model(smallx)\n",
    "print(pred.shape)\n",
    "print()\n",
    "print(pred)\n",
    "print()\n",
    "print(torch.sum(pred, dim=1))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b8236f0",
   "metadata": {},
   "source": [
    "`pred` 的每一行加总为1，其中每一个元素代表对应类别的预测概率。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ca3a05a",
   "metadata": {},
   "source": [
    "我们还可以直接打印模型对象，观察隐藏层的结构："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "20a4eb6f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MyModel(\n",
      "  (conv1): Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (conv2): Conv2d(20, 50, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (pool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (fc1): Linear(in_features=800, out_features=500, bias=True)\n",
      "  (fc2): Linear(in_features=500, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d0bc12f",
   "metadata": {},
   "source": [
    "### 2.4 定义损失函数"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffd11e33",
   "metadata": {},
   "source": [
    "对于分类问题，损失函数通常选取为负对数似然函数。在 PyTorch 中，可以使用 `torch.nn.NLLLoss` 来完成计算。其用法是先定义一个损失函数对象，然后在预测值和真实标签上调用该函数对象。注意：损失函数对象的第一个参数是预测概率的**对数值**，第二个参数是真实的标签。[文档说明](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6f4ee18e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.3102, grad_fn=<NllLossBackward0>)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lossfn = torch.nn.NLLLoss()\n",
    "lossfn(torch.log(pred), smally)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cf163e4",
   "metadata": {},
   "source": [
    "### 2.5 编写训练循环"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c53f5d1a",
   "metadata": {},
   "source": [
    "利用课上介绍的循环模板和代码示例，对模型进行迭代训练。对于本数据，选取 mini-batch 大小为200，共遍历数据3遍，优化器选为 SGD，学习率为0.001。记录每个 mini-batch 下的损失函数值存放到列表 `losses_sgd` 中，然后画出损失函数的曲线。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b99d1bb8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0, batch 0, loss = 2.309314012527466\n",
      "epoch 0, batch 20, loss = 2.3070173263549805\n",
      "epoch 0, batch 40, loss = 2.305234670639038\n",
      "epoch 0, batch 60, loss = 2.301973342895508\n",
      "epoch 0, batch 80, loss = 2.3028054237365723\n",
      "epoch 0, batch 100, loss = 2.299456834793091\n",
      "epoch 0, batch 120, loss = 2.2969160079956055\n",
      "epoch 0, batch 140, loss = 2.299391031265259\n",
      "epoch 0, batch 160, loss = 2.3002567291259766\n",
      "epoch 0, batch 180, loss = 2.297680616378784\n",
      "epoch 0, batch 200, loss = 2.2928342819213867\n",
      "epoch 0, batch 220, loss = 2.29866099357605\n",
      "epoch 0, batch 240, loss = 2.2966628074645996\n",
      "epoch 0, batch 260, loss = 2.2929019927978516\n",
      "epoch 0, batch 280, loss = 2.2988672256469727\n",
      "epoch 1, batch 0, loss = 2.289125680923462\n",
      "epoch 1, batch 20, loss = 2.295426368713379\n",
      "epoch 1, batch 40, loss = 2.297480583190918\n",
      "epoch 1, batch 60, loss = 2.2956583499908447\n",
      "epoch 1, batch 80, loss = 2.29074764251709\n",
      "epoch 1, batch 100, loss = 2.285233736038208\n",
      "epoch 1, batch 120, loss = 2.284632682800293\n",
      "epoch 1, batch 140, loss = 2.2877092361450195\n",
      "epoch 1, batch 160, loss = 2.289735794067383\n",
      "epoch 1, batch 180, loss = 2.285066843032837\n",
      "epoch 1, batch 200, loss = 2.283139228820801\n",
      "epoch 1, batch 220, loss = 2.2905187606811523\n",
      "epoch 1, batch 240, loss = 2.2848715782165527\n",
      "epoch 1, batch 260, loss = 2.281222105026245\n",
      "epoch 1, batch 280, loss = 2.2845540046691895\n",
      "epoch 2, batch 0, loss = 2.2796168327331543\n",
      "epoch 2, batch 20, loss = 2.2796127796173096\n",
      "epoch 2, batch 40, loss = 2.280289649963379\n",
      "epoch 2, batch 60, loss = 2.2780988216400146\n",
      "epoch 2, batch 80, loss = 2.2794065475463867\n",
      "epoch 2, batch 100, loss = 2.2698965072631836\n",
      "epoch 2, batch 120, loss = 2.2720232009887695\n",
      "epoch 2, batch 140, loss = 2.2733848094940186\n",
      "epoch 2, batch 160, loss = 2.2767722606658936\n",
      "epoch 2, batch 180, loss = 2.2751951217651367\n",
      "epoch 2, batch 200, loss = 2.277172803878784\n",
      "epoch 2, batch 220, loss = 2.2730095386505127\n",
      "epoch 2, batch 240, loss = 2.2703773975372314\n",
      "epoch 2, batch 260, loss = 2.2702484130859375\n",
      "epoch 2, batch 280, loss = 2.2707467079162598\n",
      "epoch 3, batch 0, loss = 2.26766300201416\n",
      "epoch 3, batch 20, loss = 2.2672176361083984\n",
      "epoch 3, batch 40, loss = 2.271594285964966\n",
      "epoch 3, batch 60, loss = 2.268263816833496\n",
      "epoch 3, batch 80, loss = 2.263566493988037\n",
      "epoch 3, batch 100, loss = 2.262798547744751\n",
      "epoch 3, batch 120, loss = 2.264540672302246\n",
      "epoch 3, batch 140, loss = 2.2666423320770264\n",
      "epoch 3, batch 160, loss = 2.257797956466675\n",
      "epoch 3, batch 180, loss = 2.258873224258423\n",
      "epoch 3, batch 200, loss = 2.261073589324951\n",
      "epoch 3, batch 220, loss = 2.254646062850952\n",
      "epoch 3, batch 240, loss = 2.254504680633545\n",
      "epoch 3, batch 260, loss = 2.252669095993042\n",
      "epoch 3, batch 280, loss = 2.256321668624878\n",
      "epoch 4, batch 0, loss = 2.247776985168457\n",
      "epoch 4, batch 20, loss = 2.251221179962158\n",
      "epoch 4, batch 40, loss = 2.2564287185668945\n",
      "epoch 4, batch 60, loss = 2.252551794052124\n",
      "epoch 4, batch 80, loss = 2.2348783016204834\n",
      "epoch 4, batch 100, loss = 2.2420458793640137\n",
      "epoch 4, batch 120, loss = 2.247875452041626\n",
      "epoch 4, batch 140, loss = 2.241600275039673\n",
      "epoch 4, batch 160, loss = 2.245187282562256\n",
      "epoch 4, batch 180, loss = 2.229278326034546\n",
      "epoch 4, batch 200, loss = 2.2348082065582275\n",
      "epoch 4, batch 220, loss = 2.231384038925171\n",
      "epoch 4, batch 240, loss = 2.2369508743286133\n",
      "epoch 4, batch 260, loss = 2.232473611831665\n",
      "epoch 4, batch 280, loss = 2.2304697036743164\n",
      "epoch 5, batch 0, loss = 2.219543218612671\n",
      "epoch 5, batch 20, loss = 2.220836639404297\n",
      "epoch 5, batch 40, loss = 2.2175159454345703\n",
      "epoch 5, batch 60, loss = 2.2104201316833496\n",
      "epoch 5, batch 80, loss = 2.217008590698242\n",
      "epoch 5, batch 100, loss = 2.2233338356018066\n",
      "epoch 5, batch 120, loss = 2.209904670715332\n",
      "epoch 5, batch 140, loss = 2.216860055923462\n",
      "epoch 5, batch 160, loss = 2.198281764984131\n",
      "epoch 5, batch 180, loss = 2.1957268714904785\n",
      "epoch 5, batch 200, loss = 2.2009117603302\n",
      "epoch 5, batch 220, loss = 2.2037200927734375\n",
      "epoch 5, batch 240, loss = 2.1987268924713135\n",
      "epoch 5, batch 260, loss = 2.1948680877685547\n",
      "epoch 5, batch 280, loss = 2.1954185962677\n",
      "epoch 6, batch 0, loss = 2.1958110332489014\n",
      "epoch 6, batch 20, loss = 2.1862263679504395\n",
      "epoch 6, batch 40, loss = 2.1812758445739746\n",
      "epoch 6, batch 60, loss = 2.1642541885375977\n",
      "epoch 6, batch 80, loss = 2.1830379962921143\n",
      "epoch 6, batch 100, loss = 2.160320520401001\n",
      "epoch 6, batch 120, loss = 2.17393159866333\n",
      "epoch 6, batch 140, loss = 2.1724579334259033\n",
      "epoch 6, batch 160, loss = 2.1707265377044678\n",
      "epoch 6, batch 180, loss = 2.153691053390503\n",
      "epoch 6, batch 200, loss = 2.1493167877197266\n",
      "epoch 6, batch 220, loss = 2.1523587703704834\n",
      "epoch 6, batch 240, loss = 2.141700506210327\n",
      "epoch 6, batch 260, loss = 2.139996290206909\n",
      "epoch 6, batch 280, loss = 2.135333299636841\n",
      "epoch 7, batch 0, loss = 2.1342813968658447\n",
      "epoch 7, batch 20, loss = 2.1308789253234863\n",
      "epoch 7, batch 40, loss = 2.1089351177215576\n",
      "epoch 7, batch 60, loss = 2.122177839279175\n",
      "epoch 7, batch 80, loss = 2.108537197113037\n",
      "epoch 7, batch 100, loss = 2.1142733097076416\n",
      "epoch 7, batch 120, loss = 2.0820465087890625\n",
      "epoch 7, batch 140, loss = 2.079834222793579\n",
      "epoch 7, batch 160, loss = 2.089418888092041\n",
      "epoch 7, batch 180, loss = 2.0532281398773193\n",
      "epoch 7, batch 200, loss = 2.0473716259002686\n",
      "epoch 7, batch 220, loss = 2.058572292327881\n",
      "epoch 7, batch 240, loss = 2.049327850341797\n",
      "epoch 7, batch 260, loss = 2.0469915866851807\n",
      "epoch 7, batch 280, loss = 2.038912057876587\n",
      "epoch 8, batch 0, loss = 2.0295090675354004\n",
      "epoch 8, batch 20, loss = 2.034121513366699\n",
      "epoch 8, batch 40, loss = 2.0445139408111572\n",
      "epoch 8, batch 60, loss = 1.9706358909606934\n",
      "epoch 8, batch 80, loss = 2.0063560009002686\n",
      "epoch 8, batch 100, loss = 1.979081630706787\n",
      "epoch 8, batch 120, loss = 1.95966374874115\n",
      "epoch 8, batch 140, loss = 1.9409905672073364\n",
      "epoch 8, batch 160, loss = 1.9092729091644287\n",
      "epoch 8, batch 180, loss = 1.9466127157211304\n",
      "epoch 8, batch 200, loss = 1.8990254402160645\n",
      "epoch 8, batch 220, loss = 1.8839267492294312\n",
      "epoch 8, batch 240, loss = 1.8403558731079102\n",
      "epoch 8, batch 260, loss = 1.883392333984375\n",
      "epoch 8, batch 280, loss = 1.8661450147628784\n",
      "epoch 9, batch 0, loss = 1.827418327331543\n",
      "epoch 9, batch 20, loss = 1.8241578340530396\n",
      "epoch 9, batch 40, loss = 1.8404518365859985\n",
      "epoch 9, batch 60, loss = 1.8025211095809937\n",
      "epoch 9, batch 80, loss = 1.7703330516815186\n",
      "epoch 9, batch 100, loss = 1.8136677742004395\n",
      "epoch 9, batch 120, loss = 1.73393976688385\n",
      "epoch 9, batch 140, loss = 1.731722354888916\n",
      "epoch 9, batch 160, loss = 1.7755272388458252\n",
      "epoch 9, batch 180, loss = 1.6819028854370117\n",
      "epoch 9, batch 200, loss = 1.643170952796936\n",
      "epoch 9, batch 220, loss = 1.6945624351501465\n",
      "epoch 9, batch 240, loss = 1.693231225013733\n",
      "epoch 9, batch 260, loss = 1.6055761575698853\n",
      "epoch 9, batch 280, loss = 1.5858447551727295\n"
     ]
    }
   ],
   "source": [
    "nepoch = 10\n",
    "batch_size = 200\n",
    "lr = 0.001\n",
    "\n",
    "np.random.seed(123)\n",
    "torch.manual_seed(123)\n",
    "\n",
    "model = MyModel()\n",
    "losses_sgd = []\n",
    "\n",
    "# 完成模型训练\n",
    "opt = torch.optim.SGD(model.parameters(), lr=lr)\n",
    "n = x.shape[0]\n",
    "obs_id = np.arange(n)  # [0, 1, ..., n-1]\n",
    "# Run the whole data set `nepoch` times\n",
    "for i in range(nepoch):\n",
    "    # Shuffle observation IDs\n",
    "    np.random.shuffle(obs_id)\n",
    "\n",
    "    # Update on mini-batches\n",
    "    for j in range(0, n, batch_size):\n",
    "        # Create mini-batch\n",
    "        x_mini_batch = x[obs_id[j:(j + batch_size)]]\n",
    "        y_mini_batch = y[obs_id[j:(j + batch_size)]]\n",
    "        # Compute loss\n",
    "        pred = model(x_mini_batch)\n",
    "        lossfn = torch.nn.NLLLoss()\n",
    "        loss = lossfn(torch.log(pred), y_mini_batch)\n",
    "        # Compute gradient and update parameters\n",
    "        opt.zero_grad()\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "        losses_sgd.append(loss.item())\n",
    "\n",
    "        if (j // batch_size) % 20 == 0:\n",
    "            print(f\"epoch {i}, batch {j // batch_size}, loss = {loss.item()}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "3e2d6346",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1d6b834b1d0>]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABGo0lEQVR4nO3deVxU5f4H8M8ZlgFZBhEREBTccEEJ9y1FMxWX9Ort2nJTbLlZLpXV70aWWbeivWubbSpZqS2uXc3UVNBcUckFxQ0FEQRRZliHZc7vD3JkBHSWM3NmmM/79ZpXc5bnOd85zcv58pxnEURRFEFEREQkE4XcARAREZFzYzJCREREsmIyQkRERLJiMkJERESyYjJCREREsmIyQkRERLJiMkJERESyYjJCREREsnKVOwBj6HQ6XLp0CT4+PhAEQe5wiIiIyAiiKKK4uBghISFQKBpv/3CIZOTSpUsICwuTOwwiIiIyQ3Z2NkJDQxs97hDJiI+PD4DaD+Pr6ytzNERERGQMjUaDsLAw/e94YxwiGbn+aMbX15fJCBERkYO5XRcLdmAlIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZMRkhIiIiWTEZISIiIlkxGSEiIiJZOXUyoi6vwoXCUqjLqvT7RFEEABRXVOHz5LPIKizTH6uu0d22zswrpVi5P8uoc4mIiMhBVu21lud/+hOb0y/X278nYTg+2XYG3+/LwsKtp/HhlGj8fDAHqReu4v9GdcbRHDX+M6EbXF0U+uTl+oqEw97bAQCoqKpB/KAIm30WIiIiR2VSy0hiYiL69OkDHx8fBAYGYuLEicjIyLhlmV27dmHQoEFo0aIFPD090blzZ3z44YcWBS2V8qqaBvcPSNyG7/dl6c+Z8d0hbD1xGUVlVXhxzVGs2J+FDvN+RXFFFR5bdhAjP0xBxU11bcsoaPS6NToRNTpRug9CRETkwExqGUlOTsbMmTPRp08fVFdXY968eRg5ciTS09Ph5eXVYBkvLy/MmjULPXr0gJeXF3bt2oXHH38cXl5e+Ne//iXJh5BL9wWb9e87v7wJAd7u+u2UUwWIeW0z7uvbBo8OjoCPhxvcXRWo0YkY8s52eCtdsfGpO+GiqG1ROVtQgh8PZOOxIe0Q4K20+WchIiKSiyBef85ghoKCAgQGBiI5ORlDhgwxutykSZPg5eWFb7/91qjzNRoNVCoV1Go1fH19zQ23nr8v2o3UC9ckq+9W7u7aCp892BMDErfhSokWAPDzjAHo1bY5BEFA9KuboS6vQmhzT2x/LhZuLg03Wr296SS2pF/GqicGQuXpZpPYiYiIzGHs77dFHVjVajUAwN/f3+gyhw8fxu7duzF06NBGz9FqtdBoNAYva5g2MNwq9TZkS/pldJz3qz4RAYC/f74HEQkbEf7CBqjLazvRXrxWjo7zftVvA8DJPA2e/P4gTl8uxqIdZ3EmvwQPfLW33jXyiytwq9zy3d9OYvaKw7c8h4iIyNbMbhkRRRETJkzAtWvXsHPnztueHxoaioKCAlRXV2PBggV4+eWXGz13wYIFePXVV+vtl7plBAC2pl/Go8tSJa1TKh5uCozpHozVh3IaPP7pAz3xY2o2kk813D9l69wh0FbrIIrAqkMXsfSP8wCAx+6MwLyxXa0VNhEREQDjW0bMTkZmzpyJDRs2YNeuXQgNDb3t+ZmZmSgpKcHevXvxwgsv4JNPPsH999/f4LlarRZa7Y0WBI1Gg7CwMKskIwCQVViGIe9uR7uWXtj2bCwAoKyyGl3n/yb5tezFhjmDsS7tEoJ8PfDw4Bujfj7ccgpn8kvw0f0xUAi1HXibudd2Ldp7rhDHL2nw8KBw/eghIiKixlg1GZk9ezbWrl2LlJQURESYPnz19ddfx7fffnvbkTjXWavPSF3ZV8vg7+UOL+WNPr3VNTp0mPcrAGDOXR3x0e+nAQCt/TwxoksgvtlzAQDQLcQXxy/deJS0+Zkh8HRzQeKvJ7DxaJ5V4pXarGEdsDYtBxevlQMAlsb3wc+HLmLDkVw8OjgCz42KROeXNwEAlsT3xo8HLqKZuwve/nsPuLkosCX9MvKLK/Bgv7ZyfgwiIrIjVklGRFHE7NmzsWbNGuzYsQMdO3Y0K7j//Oc/WLx4Mc6fP2/U+bZIRhqzJf0yfj2ai9f/FoVtJ/Pxy5+X8N690fDxcMPqQxdRUaXDA/3aAKi9P3VbDCqrdZi06A90aOmNtWmXbBq3LS2d3gfTlx4AAPw0YwD+7+cjUJdXIVjlgSXxfdDK18Ok+kSxduizayOdeImIyDFYJRl58sknsXz5cqxbtw6RkZH6/SqVCp6engCAhIQE5OTkYNmyZQCATz/9FG3atEHnzp0B1M478vTTT2P27Nl4/fXXJf0w9uxcQQkuXC1DbKeWEAQB4S9sAAD8d8odGNmtFQ5dKEJUa1+cu1KKiqoaPPDVPpkjlsadHQPwUP+2yLpahkcGR0BbrUOJthreSld4uLnoz/vjzBW89ks63pwUhZX7s7E9Ix9bnhkKv2ZufCREROSgrJKMNPajsHTpUsTHxwMA4uPjcf78eezYsQMA8PHHH+OLL75AZmYmXF1d0b59ezz22GN4/PHHoVAY95dvU0hGbnahsBSl2hp0DWn48wx+e5v+kcmbf+sOHw9XtG3RDDU6Ee9sykBZVQ3+zC7C+OgQCABGdQtCK18lTuQV4+W1x2z4SUzj5e6C0sraCeLu7BiArKtlmNInDO9savyR3UP926Jap8PcuyPR0keJH1Oz8X8/H8HQTi3x4ZQ70MzdBacvlyCqtS8TFyIiO2L1Dqy21BSTkdu5WlqJtOxrGNopUD8x2s10OhGKRo5db3lpajITxyAiYWODx96Z3APdQ1XwVroizL+ZjSMjIqKbMRlxcuvScrD33FU0b+aGz3achb+XO5Kfj8WawzmYv+54o+UiAryQeaXUhpFax9k3x0AhNN6aR0RE1sdkhPRuXswPAHaeLsAXyecwrHMg/rv1FF4e2xXa6hpMjGmNorIq3P1hMu6JDkHipB5o/2LDLRHXzRjaHp8nn7XqZ7DEqG6tsPtsIX6ZNRi56gocuViEfw1pp78fm4/nYfWhHLw9uQdUzRqf1VZbXYMDmdfQO7y5QX8XIiJqGJMRMtrNo4CA2lWHla4KCIKAGp2I4ooqqDzdoK3WYdme8xjeORD7Mq/Cz9Mdo6OCcCJXg85BPrj/q704cP4axnYPxoajuTJ9ott7a1J3TOoZCndXhcEjrczEMVhzOAdzf/wTANC+pRd+/2vumZfXHsO3ey/gbzGt8eGUO2SImojIsTAZIdkUV1TBW+lqMGrodgK8lWjbohkOXriGV8Z3xau/pFs5yloxbfxwOKvolueseKw/7r9p+v3dLwzH9ox8DO4QgKoaHd7elIE5wzuie6gKoiiioFiLwDpDmo/lqPH9vgt4ZkQng/1ERE0ZkxGyC6XaauzIKMDA9i2Qq65Al2AfaMqrsfvsFTy1Mg3xg8KRkVeMNyd1R5CvBzTlVWju5Y5zBSV4cc1R7D13VV9X99YqHM1Ry/hp6lO6KtDK1wNZV8vQyleJpOl98cOBbCTtPo9XxnfFlD5haObuqk/KhncOxJL4PjJHTURkG0xGqEl47qc/8fPBi+gRqsKPjw/AqkMXkX213K77qNxsRJdW2Hrisn571RMDIAgCerZpjoJiLaYn7ceUPm3wUP+2KCzRwq+Ze6MjqIiIHAmTEWoSqmt02Hn6CmLa+MGvmbt+36IdZ1FYWontGfm4UFhmUKZjoDdO55fot58fFYlJPVtjQOI2m8Z+O8sf64eNR3Px3d4sAMD/Zg/GuI934a7OgVhcp/Wk7mOf4ooq+HjUdrK9VFSO1Ycu4oF+beHv5S7LZyAiuhUmI+Q0rpRo8davJ/HbsTyMvyMEs4d3QLDKE2cLSlBeWYOo1ioAQOKvJ/BF8jkAwB1hfvDxcMXO01cAAC6K2o66ttS/nb/BY6i6Fk/rjeKKagyLDET0a5sB3Bh23b6lF7bOHYph7+3A+cIyPvohIrvFZIToJqIo4rfjl9E9VIXWfrXLF2gqquDuotAP1XWUyeIWT+uNR75J1W9vnHNng7P5VlTVIPNKKToH+XDOFSKyOWN/v7kSGTkNQRAwOipIn4gAgK+Hm8GcIfEDw/XvB7RrUa+Or6f2xmcP9tRvT+rZ2jrB3kbdRAQAxny0Eyv2Z9U7b+qS/YhbuBO9Xt+Khv7uSMsuwrq0HKvFSURkDLaMENWh04nILCxFuwAvFJVVIeY/W/TH0l8bhWburgCAa6WV2H/+KoZ3DkSNTkSuugJKVwU+2HIKPx+8KFf4mNwzFH/vFYrMK6W4q0sg+r35u/7YpJjW+GDKHSgs0aKovAraKh3GfLQTALAkvjeGd24lV9hE1ETxMQ2RBERRxNI/zqN7qAp9wv2NKvNI0gH8fjIfAOCtdMW3j/TF8UsafLXzXL3OtrZWd6HCm6W+NAIvrz2G0VFBmHCHPC0+RNS0MBkhsjNFZZX4auc5BKk8sWj7GVxSV8gdkoHR3YKw6XgeAOD8W2NRXaNDrrqiwUUHi8oqoS6vQtsWXrYOk4gcCJMRIjv3t8/+MJj9NeX5YQjx80CHeb/KF9RfXpvQDUm7z+NcQSkm3BGCeWO6QOnqgh2n8vHUyjT9ebtfGI6QOn1wiIjqYjJC5ADOXylF7Hs7AACHX74bzb3ccSxHjdTzV3F/vzZYn3YJwSpP/PLnJfyQmq0v5+vhCk1FtU1jjWzlg4zLxQb7Pn2gJ8b2CNZvr0vLgUIQMD46xKaxEZF9YjJC5CA2H89DZY0O43rc/gc8+2ptnxNPdxf0fn2rtUO7rZnD2kNdXoVnRnSCm6sCPRbUzolyvbNvjU5EibYaKs/GV0MmoqbL2N9vVxvGREQNGNktyOhz6/bf2PzMELz2SzruuSMEY7oHI+qV36wR3i19ur12Wv51aZfwxt+66/eXVdagmbsr7v9qL/ZnXsWU3mEI8fPEUyM6QqcTofhruvvDWddQoxPRO9wf1TU6uLpwtgEiZ8SWEaImwp4mbAtReTTYQXf1kwMxfekBTBsYjjnDO+j7x8wb0wXvb8nAkml9MLBDgK3DJSIr4aRnRE6sXUsvvHdvNAa0a4GDL43AvDFd6p3z3MhO+vetfJWSXr+xkUKTPtsNdXkVPvr9NKpqbvwd9MbGE6io0mHWisOSxkFEjoGPaYiaiH+P7oxNx/Pw71GR6B6qgo+HG/7eKxQA8MjgCNxzRwje2ZSBVYdqJ2Vr3fzGKJgtc4fq+3vYSpf5m+rt44T1RM6JyQhRE/FEbHs8Edu+wWMKhYBWvh54aWwXlFVW4x+9w9CvnT86BJ7FwPYt4Ovhhh3PxUIEEN6iGSISNto2+L8Ullbix9RsjOoWBJWnGxJWH8H+zKtYeF8MuoX44sU1R9EuwBtjewRzSDFRE8I+I0RUz2VNBcZ+tBOTe4ViXPcQTE86gCslWptdv3OQD759pB/6vHFjxNC8MV3wxsYT+u33743G5F6h+OPMFehEEXd2bGmz+IjIOBzaS0QWqTvqBQCqanToaMMJ2Z4fFYl3f8u45TntWnrhXEEpAOCTB2KMGh5NRLbDZISIJHfg/FWcyNVg4dbTKCytlDuces6/NVbuEIioDo6mISLJ9Qn3x9QB4fjioV7w93LHh1OiDY7PH9dVpshqPfHdQWgqqurtv1BYijP5JTJERETGYAdWIjJZ73B/HHxpBARBQEtvD7y/JQOJk7qjc5AvXvtfuv68vhH+2J951WZx/XosD0pXBfpE+KNLsC/cXRToEOiNoe/uAFC7MnGAt7TDmInIcnxMQ0SS6vvGVuQXa/Hv0Z3xRGx7zFx+CBuO5GJJfG8MbB+ALemXkbT7PA5euGaTeKb0DjNY12f5Y/0wsP2NidVeXHMUpy8XY8Vj/TkDLJHE2GeEiGSRr6nAvsyrGB0VBDcXBURRRGllDbyVhg2xdWeM3TBnMMZ+tMtmMc65qyPm3t3JII7vHumHwR05+yuRlNhnhIhkEejrgfHRIXD7q5VBEIR6icjNIlv5GGw/2K+N1eIDgI9+P43xH+/C3nOF+n019v93GVGTxWSEiGSxbuYgvDS2C/58ZSRcXRRYeN8daNuiGTbMGYxglYfVr380R437vtyr376sqcCawxdRVGZ/o4SImjqTHtMkJiZi9erVOHnyJDw9PTFw4EC8/fbbiIyMbLTM6tWrsWjRIqSlpUGr1aJbt25YsGABRo0aZXSQfExD5Fw0FVV44Ku9GN0tCPvPX0PKqQKbXv/+vm1wOOsa3pzUHT3bNLfptYmaEqv0GRk9ejTuu+8+9OnTB9XV1Zg3bx6OHj2K9PR0eHl5NVjm6aefRkhICIYNGwY/Pz8sXboU7733Hvbt24eYmBhJPwwRNT0VVTX4cOsptPByR7cQFR78ep9Nr9/Q3CU1OhH//Hofwvw98c7foxsoRUSAjTqwFhQUIDAwEMnJyRgyZIjR5bp164YpU6Zg/vz5Rp3PZISIrtuekY/pSw8AADzcFKio0ln1emfeiENVjQilqwIKhYA739mG7Kvl+uNDOrXE/HFd4enugtZcL4fIgLG/3xbNM6JWqwEA/v7+RpfR6XQoLi6+ZRmtVgut9sY6GBqNxvwgiahJGRYZiM8e7In/HbmEf4/ujPKqGnyRfA5PxLbHf/6Xjp2nr0h6vZUHsvHmxhPw83RDqH8zg0QEAFJOFWDEB8kAgI/uj8E90ZySnshUZreMiKKICRMm4Nq1a9i5c6fR5d5991289dZbOHHiBAIDAxs8Z8GCBXj11Vfr7WfLCBHdyivrjuGbPRdku36HQG9snTtUtusT2RurD+2dNWsWjhw5ghUrVhhdZsWKFViwYAF++OGHRhMRAEhISIBarda/srOzGz2XiOg6QbixsN+J10ZjT8Jwm17/TH4JcorKb38iERkwKxmZPXs21q9fj+3btyM0NNSoMj/88AMeeeQR/PjjjxgxYsQtz1UqlfD19TV4ERHdTp1cBJ7uLghWeSL1pRE4/PLdNoth0FvbsHxfFgCgukaHy5oKm12byFGZlIyIoohZs2Zh9erV2LZtGyIiIowqt2LFCsTHx2P58uUYO5arahKRdQgQ6u0L8FaiuZc7/jOhm35fl2BfjOkepN++edI1S7245ihO5Gow9N0d6Pfm7zhysUjS+omaGpM6sM6cORPLly/HunXr4OPjg7y8PACASqWCp2dtL/KEhATk5ORg2bJlAGoTkalTp2LhwoXo37+/voynpydUKpWUn4WInNzoqCAs+SMTLX3qL4b3YL+2yLpahsggX0zu2RqCIKCwRItm7q7wdHdBUVklLhVVYMxHxveBu5W4hTfq+Wz7WdzXNwyebi7o166FJPUTNSUmdWCt+zy2rqVLlyI+Ph4AEB8fj/Pnz2PHjh0AgNjYWCQnJ9crM23aNCQlJRl1XQ7tJSJjpV/SoHVzT6g83cwqX3fNHGvITBzT6L+lRE2NVYb2GpO33JxgXE9KiIhsoWuIff/BUlmjg9LVpdHju05fQaCvEp0kfnREZM+4Ng0RUR2bnr4TXzzUC9MHhVul/ovXakfbHDh/Fd/uOW/wR96Z/GL8c/E+jPwwxSrXJrJXTEaIiOroHOSLUd2CEBVinT5tj397EABw7+d78PK64waTtJ3MK653/se/n8aw93bgSom23jGipoLJCBFRA3qEGiYjAd7uaOHlrt8+++YYzBrWweR6z+SXYF1ajn771OVirD50EZeKylGjq/8o/P0tp5B5pRSfbT9r8rWIHIVF08ETETVVHVv54KcZA9DCyx0qTze08FaivLIGi3edw91dg+CiEDBreAd8sv2MyXU/tTJN//71DScAAJ1aeeOxO9s1WmbricuYEdsOgT4eJl+PyN6xZYSIqBF9wv3RrqU3WnjXDhX2dHfBrOEdERlU27nUw80Fu18YjiGdWlp8rVOXS/D8z0f02xl5xSjVVuu3s66WYfDb2+uV234yHw8t3odLnPmVHJhFq/baCof2EpG9s/aQ4OtuHhp8/bqxkS2RNL2vTWIgMpbV16YhIiLb+/1EfoP7L2vYwZUcF5MRIiIHknG5GDqdCN1NnV0doJGbqFHswEpEJIF5Y7pg2d7z+L9RndEh0NtgOngprTmcg3d/ywAAzBzWXr+/oWHBRI6CfUaIiKzgWI4aZZU1WP9nDr7bm2WTa678V3/sPF2AuXdHwkXBKedJfuwzQkQko6jWKvSN8Ier4sY/s/+dcodVr3nfl3vx6fazaP/iRqteh0hqTEaIiKzoyWHtEeTrgTnDO2BiTGu5wyGyS0xGiIisKNDHA3sShmPuyEgAQFRrw6bqAe1aWOW6hSValGiroS6vwqqDF1FSZ84SInvDDqxERFZWd16Q1U8MwrWySvR783cAsFrfjl6vb4WrQkD1X6NutqRfxucP9bLKtYgsxZYRIiIbcndVoJXvjSndBQH4cEp0vbVwpFBdZ/jvpuN5tzy3slon+fWJjMVkhIhIRoIg4G8xoVg/azD+0TvU4FjdobtSqKzWYX/mVVTV6PBTajb+OFO7YvB/t55C5Mu/Ii27SNLrERmLj2mIiGQwtkcwNhzJxYwhNxbHe/Nv3dHK1wMfb6tdfO/5UZ3RI9QPj397UJJrvrz2GH5IzUbfCH/sz7wKAHh8aDt8kXwOAPDaL8ex+slBklyLyBScZ4SISAY6nYirZZUI+GsRvutEUcTy/VmIDvVDVOvaRze2Wvemb4Q/fnx8gE2uRc7B2N9vtowQEclAoRDqJSJA7WObB/u1lSEiwN2FT+5JHvzmERHZOS93F5tcx9WFs7aSPJiMEBHZOddGWix+mTVY0uuoy6vw4ZZTKKvknCRkW0xGiIjsnFudZCQiwEv/vnuoCvdEh0h2ncNZRVj4+2l0nf+bZHUSGYPJCBGRnbu+ps38cV1RrTOcD2ThfXdY9drllTVWrZ8IYDJCRGT3BncMwKnX4/Dw4Ai8+bfuAIDnRnYCYDi7q5Q+3HIKqw5eRJf5m7B8X/1Vh0VRhKaiyirXJufDob1ERA6mvLIGnnU6tc79IQ2rD+dY9Zrn3xprsD1r+SH870gufpk1GN2tMHssNQ3G/n6zZYSIyMF43jS65t17o61+zYoqw8c1/zuSCwD4auc5q1+bmj4mI0REDs5FISD5+Vg8OjhCvy+ozvo3Upj02e4G99t90zo5BCYjRERNQNsWXnhpXFf99pBOAXhtQjfJ6k/P1WDuj2m4UqI12O8AT/rJATAZISJqgkQRmDogvMHF9lwU5nV6XX0oB71f34qconJLwyMywGSEiKgJiYsKAgDEDwoHAMy5qyPevzcar95T20oS4O2Os2+OweZnhph9jUFvbdO/Z7sISYFr0xARNSGfPdgTmopqqDzdAABKVxdM7hUKURQREeCFzsE+AGpbTiTBbIQkYFLLSGJiIvr06QMfHx8EBgZi4sSJyMjIuGWZ3NxcPPDAA4iMjIRCocDTTz9tSbxERHQLgiDoE5Gb9w/p1BKBPrUdW0WJsgip6iHnZlIykpycjJkzZ2Lv3r3YsmULqqurMXLkSJSWljZaRqvVomXLlpg3bx6io60//IyIiG7vpolczdZQC8vV0krU6JikkPFMekyzadMmg+2lS5ciMDAQBw8exJAhDT9/DA8Px8KFCwEAS5YsMTNMIiKSkk6i5zQ3V3P8khpjP9qFOzsG4NtH+klyDWr6LOozolarAQD+/v6SBHOdVquFVntj+JhGo5G0fiIiZ9fsponTpPL9X1PH7zx9BbnqcgSrPK1yHWpazB5NI4oi5s6di8GDByMqKkrKmJCYmAiVSqV/hYWFSVo/EZGza9fSG7OHd7C4nk3H83AyT4MvU86iqsbw2c+AxG0ovGleEqKGmJ2MzJo1C0eOHMGKFSukjAcAkJCQALVarX9lZ2dLfg0iImf37MhISeoZ/d+deHPjSXSc9yvOXzHsQ5iRVyzJNahpMysZmT17NtavX4/t27cjNDRU6pigVCrh6+tr8CIiIukte7ivpPXtPltosK1043RWdHsmfUtEUcSsWbOwevVqbNu2DREREbcvREREdmtIp5boE95cv/1kbP0ZWy2Rp9bi3s9341/LUiWtl5oWkzqwzpw5E8uXL8e6devg4+ODvLw8AIBKpYKnZ20npYSEBOTk5GDZsmX6cmlpaQCAkpISFBQUIC0tDe7u7ujatWu9axARkW3VHYbbvJm7pHXPXH5I/14URQiCeVPRU9NmUjKyaNEiAEBsbKzB/qVLlyI+Ph5A7SRnWVlZBsdjYmL07w8ePIjly5ejbdu2OH/+vOkRExGRpAZ1CMChrCI0c3cxGPK74rH+2JdZCE83FyT+etLi69ToRLi6MBmh+kxKRoxZnTEpKcmsckREJI+ZwzogSOWBoZ1a4mzBjQ6oA9q3wID2LXDwwjVJrlMjilyDhBrE7wURkZPzcHPBg/3aAgBa+3nig39Eo3PQjYEDPdv4YWyPYGw4kmvRdXafLcSwyECL6qCmid2ciYhITxAETOoZiq4hvgb7Pn2gJ+7va9mcT9OXHrA0PGqimIwQEZFR3pjYHcnPx2LFY/3lDoWaGCYjRERkFIVCQNsWXrBkQMzNs7QSAUxGiIjIRJaMSbj/y70G22WV1Zi+dD9W7s9qpAQ5AyYjRERksc//2cuo81JvGpmTtPs8tmcU4IXVR60RFjkIJiNERGSShqZ4Hx0VZHT5uT+k4cnvD6KqRofiimopQyMHxaG9RERkEh/ljZ+O9NdGoZm7aT8lqw/nAABGdcsFp0AjgC0jRERkovAALwT6KNEh0Buebi5m1/PUyjRU68zvgKKtrsFlTYXZ5cl+sGWEiIhM4uaiwB8vDAcAi9ea+TLlnP59UVklfD3coFAYV+fYj3bhTH4JNj8zBJ1a+VgUB8mLLSNERGQyNxcF3FwMf0L+NaSdRXXe8doWtHtxI3Zk5Bt1/pn8EgDAxqOWzQxL8mMyQkREknhxTBdJ6olfegDv/nYSL689ZtT5AnueODw+piEiIrvz6fazAICi8iqMiQpCXPfgRs+18EkR2QG2jBARkVWceSPO4jp++fMSnvj+0C3P+Tz5rMXXIXkxGSEiIsl89mBPtPbzxM8zBsDVRbqfmJpbjLopq6yR7DokDyYjREQkmTHdg/HHC8PRO9wfADC4Q4Ak9bZ/cSMKS7QAgG0nLyNh9RFJ6iX7wGSEiIis5puH++KNv0VJUteQd7YDAB5OSsWK/dmS1En2gR1YiYjIalwUAh7o2wYdWnqjTYtmGJC4zey6Svk4psliMkJERFYlCAL6tWshdxhkx/iYhoiIHIYomj99PNkvJiNERGQzk3uGWlR+c/pliSIhe8JkhIiIbOa5UZ0Q8ddCe+b4bPsZiSMie8BkhIiIbCZY5Yntz8Xix8cHmFX+z4vq256jra7BsRw1H+k4ECYjRETUpDz6TSrGfbwL3+/LkjsUMhKTESIicnhrD+dg1+krAICdf/332z0X5AyJTMBkhIiI7MaILoFmlXv6hzT8c/G+Bo+991sGHlq8D1U1OktCIytiMkJERDZ3c2+OHc/FYuaw9nh7cg+L6m1oDZtPtp/BztNXsO1kvkV1k/Vw0jMiIpLVcyM7ITzAC8+P6mxxXQMSf9e/v6QuNzhWXcMOrfaKLSNERGRz3sobfwvPGNpesnrzi7X698UV1QbHJFxEmCTGlhEiIrK5lj5KvDO5BzzcXeBqxSzhvi/36N8rBMFq1yHLMBkhIiJZ/KNPmNWvsffcVf17FwWTEXtlUjqamJiIPn36wMfHB4GBgZg4cSIyMjJuWy45ORm9evWCh4cH2rVrh88//9zsgImIiMyhYDJit0xKRpKTkzFz5kzs3bsXW7ZsQXV1NUaOHInS0tJGy2RmZmLMmDG48847cfjwYbz44ouYM2cOVq1aZXHwRETU9HRvrbJKvS58TGO3BNGC+XILCgoQGBiI5ORkDBkypMFz/v3vf2P9+vU4ceKEft+MGTPw559/Ys+ePQ2WuZlGo4FKpYJarYavr6+54RIRkQOo0YnQVteg6/zfJK33u0f6YXDHAEnrpFsz9vfbol5DanXtGgH+/v6NnrNnzx6MHDnSYN+oUaOQmpqKqqqqBstotVpoNBqDFxEROQcXhYBm7tJ3aVRwNI3dMvt/jSiKmDt3LgYPHoyoqKhGz8vLy0OrVq0M9rVq1QrV1dW4cuVKg2USExOhUqn0r7Aw63dyIiIi+9TSR4m3J3e3uJ66j2kKS7TQVDT8BzHZntnJyKxZs3DkyBGsWLHitucKNz2nu/5k6Ob91yUkJECtVutf2dnZ5oZJREQOzkUQEBtp3jTxdV3/zSmuqEKv17eix4LNFtdJ0jCrHWz27NlYv349UlJSEBoaestzg4KCkJeXZ7AvPz8frq6uaNGiRYNllEollEqlOaEREVET46IQIEXf0+t/CJ8raHzQBcnDpJYRURQxa9YsrF69Gtu2bUNERMRtywwYMABbtmwx2Ld582b07t0bbm5upkVLRERORxCAAC/L/0C9vmyNrs64DV0Da9mQ7ZmUjMycORPfffcdli9fDh8fH+Tl5SEvLw/l5Tfm/09ISMDUqVP12zNmzMCFCxcwd+5cnDhxAkuWLMHixYvx3HPPSfcpiIioybm3V23L+zMjOkkyR8hbv9aO6qybf1QzGbELJj2mWbRoEQAgNjbWYP/SpUsRHx8PAMjNzUVWVpb+WEREBDZu3IhnnnkGn376KUJCQvDRRx9h8uTJlkVORERN2tuTe+DpuzuhtZ+nJPX9eVGNa6WVqDujhc782S1IQiYlI8ZMSZKUlFRv39ChQ3Ho0CFTLkVERE5OoRAkS0SuS/z1BP7e68YITbaM2AeOuiYiIofQzN3F4jqyr5ajpk4CUlNT+96C+T9JAkxGiIjIIaybOQgT7gjBisf6m12HIBg+mqnW6XChsBR93vgdi3aclSJMMgNX7SUiIofQsZUPFt4XA3WZ+ZOV7T5biKullfrtGlHE25tO4kqJFm9vOoknYttLESqZiC0jRETkVE7mFevf1+hE8AmN/JiMEBGR06quEaHgar6yYzJCREQORYR0TRk6UZRkdleyDJMRIiJyKHUfqzw+pB3Wzxpkdl3VOraM2AMmI0RE5LAevbMdeoT6oVfb5maVr9GJyCkqv/2JZFUcTUNERA7F19MN3kpX6EQRzZvVrnFm7mzx//lfOg5euCZhdGQOJiNERORQXBQCDr48AqIIuLrUNvBP7hmKA+dNTyp2nr4idXhkBiYjRETkcJSuhrOx/qN3GNoHeqOwpBIzvjtodr3VNTp9gkO2wztOREQOT6EQ0CfcH74elv2NncpHNrJgMkJERE1GsIUL69335V6JIiFTMBkhIqImIyLAC3eE+em3B3VoIV8wZDQmI0RE1KSM6xGsfy/AvGE26rIqg9V9ybqYjBARUZNyzx0hAIA7OwaYNbvq6cvFiH5tM9q/uBHVNTqJo6OGMBkhIqImJdDHAyf/MxrLHu6LV8Z3Q/NmbnghrrPR5Zfvz9K/P35JAwAo1VajvLJG8lipliCK9r9eoUajgUqlglqthq+vr9zhEBGRA9HpRCgUAsJf2GByWReFgC8f6oVHvkmFu4sCGa+PhsDp441m7O83W0aIiKhJU/w1PeuwyJYml63RiXjkm1QAQGWNDtXsR2IVTEaIiMgpSLEgHju1WgeTESIicjqebi63P6kBOvvv2eCQmIwQEZHTeWxIO7PK8TGNdTAZISIip/PUXR3NKqdjMmIVTEaIiMgp1E0jXBTm9R+p0YnYcCQXm4/nSRMUAeCqvUREREYrLK3EzOWHAACnXo+Duyv/ppcC7yIREZGRNOVV+vdVnJ1VMkxGiIiIjFT3Uc+tRtaUVVazf4kJmIwQEZFTkGLC8bpV6BppGLlSokXX+b9h0qLdFl/PWTAZISIiAvBkbPvbnpOnqdC/r24kG/n9xGUAQFp2kSRxOQMmI0RERADu6hJ423PmrDisf19Up/8IWcbkZCQlJQXjx49HSEgIBEHA2rVrb1vm008/RZcuXeDp6YnIyEgsW7bMnFiJiIjM5uZy65+8Xm39TarvrveTG9wvgAvpmcrkZKS0tBTR0dH45JNPjDp/0aJFSEhIwIIFC3D8+HG8+uqrmDlzJn755ReTgyUiIjLXS2O7IrS5JxaM7woAWPZwX/2xYJWHdBdiLmIyk+cZiYuLQ1xcnNHnf/vtt3j88ccxZcoUAEC7du2wd+9evP322xg/fryplyciIjJLmxbNsOvfw/XbQzq1xFdTe2PpH5l4/x/RZtVZXlkDQQA86qx1w1zEdFaf9Eyr1cLDwzDj9PT0xP79+1FVVQU3Nzdrh0BERNSgu7u2wt1dW5ldvtsrm6B0dUHK/w1DWWU12rbwgiDB6sDOxuodWEeNGoWvv/4aBw8ehCiKSE1NxZIlS1BVVYUrV640WEar1UKj0Ri8iIiI7I1OBMqratDnja0Y+u4OFBRr5Q7JIVk9GXn55ZcRFxeH/v37w83NDRMmTEB8fDwAwMWl4SWcExMToVKp9K+wsDBrh0lERGSxk3kaPqYxg9WTEU9PTyxZsgRlZWU4f/48srKyEB4eDh8fHwQEBDRYJiEhAWq1Wv/Kzs62dphEREQWE0WAT2lMZ7OF8tzc3BAaGgoAWLlyJcaNGweFouFcSKlUQqlU2io0IiIiSew8XQAFsxGTmZyMlJSU4MyZM/rtzMxMpKWlwd/fH23atEFCQgJycnL0c4mcOnUK+/fvR79+/XDt2jV88MEHOHbsGL755hvpPgUREZEd+GpnpsH2yTwNOgf5yhSN4zA5GUlNTcWwYcP023PnzgUATJs2DUlJScjNzUVWVpb+eE1NDd5//31kZGTAzc0Nw4YNw+7duxEeHm559ERERHbs6EU1kxEjmJyMxMbG3nKxoaSkJIPtLl264PDhww2fTEREZEcE4cZieAvGd8WCX9Itqo+PbIzDtWmIiIj+Ujd1mDYw3OL6GukaSTfhbSIiIvpL3QnLBEHAK39NHW92fRzoaxQmI0RERI2YcEdri8r/8ucliSJp2piMEBER/eXmdgxPt4Yn5zTW7yfzLSrvLJiMEBER/cVLaTiuw8ONP5O2wLtMRET0l6TpfdC2RTN8PbU3gNp+I+OjQxDTxk+S+ku11Xg46QB+PMCZxeuy2QysRERE9i6mTXMkPz/MYN/H98cAAMJf2GBx/V/tPIdtJ/Ox7WQ+/tGH665dx5YRIiIiGykqq5I7BLvEZISIiIhkxWSEiIjIijYdy9XPXF53BnOdrvHZzJ0NkxEiIiIrmvHdIfxyJBcAUDf/mL2CS6Vcx2SEiIjIyj7YnIHiiiqIuJGNbDiaK2NE9oXJCBERkZWdLyxD/NID4JOZhnFoLxERkQ0cvHANBy9ckzsMu8SWESIiIpIVkxEiIiKSFZMRIiIimZwrKJE7BLvAZISIiEgmZwtK5Q7BLjAZISIiMoKXu4vkdbq6CJLX6YiYjBARERlh67NDDbalSE7cFPwZBpiMEBERGSVY5YnBHQL020cXjMKWZ4bA3dX8n1K2jNRiMkJERGQGhUJAx1Y+mD4w3Ow6XBVMRgAmI0RERBYRBPMTiud++pMToYHJCBERkdH+3isUANAtxFe/z5LGjfOFZZi8aLelYTk8TgdPRERkpAl3hKBdSy90CPTW77OgYYT+wpYRIiIiIwmCgB6hfmjmfuNveYUE2cjUJfvx1MrD0DnpSnpsGSEiIrJAWPNmFteRcqoAADCiSyuMjw6xuD5Hw5YRIiIiC0zq2VqyuvKLtZLV5UiYjBAREVnA1UWBEV0CJanrp9RsVFTVSFKXI2EyQkREZCdO5hXjgy2n5A7D5piMEBER2ZFtJ/PlDsHmTE5GUlJSMH78eISEhEAQBKxdu/a2Zb7//ntER0ejWbNmCA4OxvTp01FYWGhOvERERHZHlHAQjDNOympyMlJaWoro6Gh88sknRp2/a9cuTJ06FY888giOHz+On376CQcOHMCjjz5qcrBERERNnRRDhR2NyUN74+LiEBcXZ/T5e/fuRXh4OObMmQMAiIiIwOOPP4533nnH1EsTERE1eZZML++orN5nZODAgbh48SI2btwIURRx+fJl/Pzzzxg7dmyjZbRaLTQajcGLiIjIXs0c3gEA0LONHz6cEm1RXc6Xithg0rOBAwfi+++/x5QpU1BRUYHq6mrcc889+Pjjjxstk5iYiFdffdXaoREREUmiZ5vmOLpgJLyVrhAEAX7N3DF96QGz6lI44dASq3/k9PR0zJkzB/Pnz8fBgwexadMmZGZmYsaMGY2WSUhIgFqt1r+ys7OtHSYREZFFfDzcJHnEohAEvP6/dDz6zQGnmR7e6i0jiYmJGDRoEJ5//nkAQI8ePeDl5YU777wTr7/+OoKDg+uVUSqVUCqV1g6NiIjIKjoH+ZhdVhAEfL0rEwCQeuEa+kb4SxWW3bJ6y0hZWRkUN7U5ubi4AABEKcdCERER2YlglSd+e3qIWWXrtq1UVuukCcjOmZyMlJSUIC0tDWlpaQCAzMxMpKWlISsrC0DtI5apU6fqzx8/fjxWr16NRYsW4dy5c/jjjz8wZ84c9O3bFyEhzrcYEBEROYdIM1tHqmpuJCA6J/mj3eTHNKmpqRg2bJh+e+7cuQCAadOmISkpCbm5ufrEBADi4+NRXFyMTz75BM8++yz8/PwwfPhwvP322xKET0RE1LQcv3RjBGlZZbWMkdiOIDrAsxKNRgOVSgW1Wg1fX1+5wyEiIjJK+Asb9O/9mrmhqKzK5DoOvXw3/L3cpQzLZoz9/XbCAURERES20SHQGwAgCMC/R3c2q44dGU1/rRomI0RERFby9dTeGN0tCOtmDoKLmYvOzP3xT4z6MAVXSysljs5+MBkhIiKykvAAL3z+UC/0CPVDgLf5j1oyLhfji+SzuFBYio9/Pw11uemPe+yZ1ecZISIiImBYZCDiB4YjMsgHCauPmly+skaHcR/tQrG2GqfzS/DR/TFWiFIeTEaIiIhsQBAELLinGwDg9xP52Hrissl1FGtrR9fsPVcoaWxy42MaIiIiG7N01vimNks8kxEiIiIbMycXEeqUcoBZOUzCZISIiMjGzGkZ2Xm6QP++sImNrGEyQkRE5ABO55fIHYLVMBkhIiKyMcGsBzVNF5MRIiIikhWTESIiIhuzdDRNU8NkhIiIyMaYjBhiMkJERGRj7DNiiMkIERGRrTEXMcBkhIiIiGTFZISIiIhkxWSEiIjIxqR6SqNrIovUMBkhIiKyMaHOcJqpA9qaVcd7v2WgzxtbcamoXKqwZMNkhIiIyMbqtoy8NiEKjw9pZ3Idn2w/g8LSSny87Yx0gcmEyQgREZGN1ZtnxMlH1zAZISIisrHZwzsAAKb99YjGsnlHRJwtKIEoOm7/EUF0gOg1Gg1UKhXUajV8fX3lDoeIiMhildU6uLvWtglk5BVj1H9TLKrv8SHtkDCmixShScbY32+2jBAREcngeiICAJFBPvjhX/0tqu+LlHOWhiQbJiNERER2INDXQ+4QZMNkhIiIyA64OPHqeUxGiIiI7ICqmZvcIciGyQgREZEdUHm6YdnDffH8qEi5Q7E5V7kDICIiolpDOrWEt4fz/TSzZYSIiIhkxWSEiIjIjvha0DJSqq3GurQcaCqqJIzI+kxORlJSUjB+/HiEhIRAEASsXbv2lufHx8dDEIR6r27dupkbMxERUZPVIdDH7LL/XnUET61Mw+zlhyWMyPpMTkZKS0sRHR2NTz75xKjzFy5ciNzcXP0rOzsb/v7+uPfee00OloiIyBm08lWaVe5/R3IBAMmnCqQMx+pMbguKi4tDXFyc0eerVCqoVCr99tq1a3Ht2jVMnz7d1EsTERE5hWUP97N4engA2HAkF+1aeqFLsH0vpWLzPiOLFy/GiBEj0LZt20bP0Wq10Gg0Bi8iIiJnERnkg8//2dOiOvaeK8TM5YcQt3CnRFFZj02TkdzcXPz666949NFHb3leYmKivkVFpVIhLCzMRhESERHZC8tmZD2Z6zh/yNs0GUlKSoKfnx8mTpx4y/MSEhKgVqv1r+zsbNsESEREZCcsnR3+3JVS/XtRFC2MxrpsloyIooglS5bgoYcegru7+y3PVSqV8PX1NXgRERE5E0tXqlm254L+vb2v6GuzZCQ5ORlnzpzBI488YqtLEhEROayuIdL9If7ptjOS1WUNJo+mKSkpwZkzNz5UZmYm0tLS4O/vjzZt2iAhIQE5OTlYtmyZQbnFixejX79+iIqKsjxqIiKiJi60eTPpKrPzBYFNbhlJTU1FTEwMYmJiAABz585FTEwM5s+fD6C2k2pWVpZBGbVajVWrVrFVhIiIyExdLRiea+e5iOktI7GxsbfsCJOUlFRvn0qlQllZmamXIiIicmpJ0/vg36uO4L17o/FndhHSzRwhI1jaG9bKnG9pQCIiIgcRGxmIfS+OAAAczioyux47z0W4UB4REZEj6B3e3Oyydp6LMBkhIiJyBAPbB8BHad4DjbqPaWp0IrakX0Z+cYVUoVmMyQgREZGDGNwxwKxydVtGVh7IwmPLUnH3B5avfSMVJiNEREQOQmfmTKp1+4xsO5EPAFCXV0kRkiSYjBARETmIfw1pZ2bJ2mzk0W8O4PeT+dIFJBEmI0RERA6iV1t/bH8u1uRyglDbV2TrCftLRAAmI0RERA4lIsDL5KG6Aux7sTwmI0RERA7mm+l9TTpfEAD7TUWYjBARETmcMH/T1q0p1dbAjhtGmIwQERE5GlMfuZRoq1FZo7NSNJZjMkJERNTE9Wzjh35vbJU7jEYxGSEiInIwpj5xUZdXobSypsFj2uqG99sSkxEiIqImrrFVez/bcQaRL21CyqkCG0dkiMkIERGRgzG1M2p5I60i72zKAAAkrD5qaUgWYTJCRETkcEzLRnKKym953NR5S6TGZISIiMjJKWTORpiMEBERORip5wxhywgRERHJii0jREREZBKpJ1NlywgRERGZxM/TzWA7+flYi+qTORdhMkJERORoAn098NqEbogOVWHn/w2Dn6e7RfXJ/ZjGVdarExERkVmmDgjH1AHhAABNRZVFdfExDREREVnE0lxC7pYRJiNEREQkKyYjRERETo4tI0RERCQrhczZAJMRIiIiB9fYqrxGl5d5cC+TESIiIgfn5e5isH3s1VEmledoGiIiIrKIIAho39JLv+3mYlp2YWnLiqVMTkZSUlIwfvx4hISEQBAErF279rZltFot5s2bh7Zt20KpVKJ9+/ZYsmSJOfESERFRA6b0CQMA9G7b3OQOqQqZW0ZMnvSstLQU0dHRmD59OiZPnmxUmX/84x+4fPkyFi9ejA4dOiA/Px/V1dUmB0tEREQNe2RwO0SH+iGqtcqMZMTBZmCNi4tDXFyc0edv2rQJycnJOHfuHPz9/QEA4eHhpl6WiIiIbsFFIaBfuxYAAFE0bSm9Jr82zfr169G7d2+88847aN26NTp16oTnnnsO5eXl1r40ERGRUzK1D4jcHVitvjbNuXPnsGvXLnh4eGDNmjW4cuUKnnzySVy9erXRfiNarRZarVa/rdForB0mERGR0zpw/hqqanRwc5FnXIvVr6rT6SAIAr7//nv07dsXY8aMwQcffICkpKRGW0cSExOhUqn0r7CwMGuHSURE5NR+Sr0o27WtnowEBwejdevWUKlU+n1dunSBKIq4eLHhD56QkAC1Wq1/ZWdnWztMIiIikonVk5FBgwbh0qVLKCkp0e87deoUFAoFQkNDGyyjVCrh6+tr8CIiIiLr6dnWT7Zrm5yMlJSUIC0tDWlpaQCAzMxMpKWlISsrC0Btq8bUqVP15z/wwANo0aIFpk+fjvT0dKSkpOD555/Hww8/DE9PT2k+BRERERn49IGeJp0v5/Bek5OR1NRUxMTEICYmBgAwd+5cxMTEYP78+QCA3NxcfWICAN7e3tiyZQuKiorQu3dvPPjggxg/fjw++ugjiT4CERER3Wxsj2BM7tnwE4iGyDnxmcmjaWJjY285fjkpKanevs6dO2PLli2mXoqIiIhsRM4p4bk2DRERURPlrXS5/Ul/cajHNEREROQY5tzV0ehz5XxMw2SEiIioiWrhrUSXYONGpLJlhIiIiKxiSMcAo86Tc0p4JiNERERN2DN3d8LrE6Nuex5bRoiIiMgqPNxc8M/+bW97HpMRIiIikhU7sBIREZGsOM8IERERyYotI0RERCQr9hkhIiIiWTEZISIiIlkJMmYETEaIiIiILSNEREQkL3ZgJSIiIlmxZYSIiIhsYnjnwAb3c20aIiIisolm7i4N7mfLCBEREVlVSx8lAGBUt6AGjzMZISIiIqva/PQQrPxXf4zrEYx2Lb3qHWcHViIiIrKq5l7u6N+uBQRBwC+zBuOrqb0NjnNtGiIiIrIZL6Ur7u7aCi/EdZY7FABMRoiIiJyWu4t9pAH2EQURERHZnJzDeetiMkJEROSk7CQXYTJCRETkrOTstFoXkxEiIiInZSe5CJMRIiIikheTESIiIidlJw0jTEaIiIiclp08p2EyQkRE5KTsIxVhMkJEROS07KRhxPRkJCUlBePHj0dISAgEQcDatWtvef6OHTsgCEK918mTJ82NmYiIiCQg2EnbiKupBUpLSxEdHY3p06dj8uTJRpfLyMiAr6+vfrtly5amXpqIiIgkZC8tIyYnI3FxcYiLizP5QoGBgfDz8zO5HBEREVlHx0BvuUMAYEYyYq6YmBhUVFSga9eueOmllzBs2LBGz9VqtdBqtfptjUZjixCJiIicSu9wf/x3yh2ICPCSNQ6rd2ANDg7Gl19+iVWrVmH16tWIjIzEXXfdhZSUlEbLJCYmQqVS6V9hYWHWDpOIiMgpTYxpjegwP1ljEERRFM0uLAhYs2YNJk6caFK58ePHQxAErF+/vsHjDbWMhIWFQa1WG/Q7ISIiIvul0WigUqlu+/sty9De/v374/Tp040eVyqV8PX1NXgRERFR0yRLMnL48GEEBwfLcWkiIiKyMyZ3YC0pKcGZM2f025mZmUhLS4O/vz/atGmDhIQE5OTkYNmyZQCA//73vwgPD0e3bt1QWVmJ7777DqtWrcKqVauk+xRERETksExORlJTUw1GwsydOxcAMG3aNCQlJSE3NxdZWVn645WVlXjuueeQk5MDT09PdOvWDRs2bMCYMWMkCJ+IiIgcnUUdWG3F2A4wREREZD/sugMrERER0XVMRoiIiEhWTEaIiIhIVkxGiIiISFZMRoiIiEhWTEaIiIhIVkxGiIiISFYmT3omh+tToWg0GpkjISIiImNd/92+3ZRmDpGMFBcXAwDCwsJkjoSIiIhMVVxcDJVK1ehxh5iBVafT4dKlS/Dx8YEgCJLVq9FoEBYWhuzsbM7sagTeL+PxXhmP98p4vFfG470ynjXvlSiKKC4uRkhICBSKxnuGOETLiEKhQGhoqNXq9/X15ZfVBLxfxuO9Mh7vlfF4r4zHe2U8a92rW7WIXMcOrERERCQrJiNEREQkK6dORpRKJV555RUolUq5Q3EIvF/G470yHu+V8XivjMd7ZTx7uFcO0YGViIiImi6nbhkhIiIi+TEZISIiIlkxGSEiIiJZMRkhIiIiWTl1MvLZZ58hIiICHh4e6NWrF3bu3Cl3SDa1YMECCIJg8AoKCtIfF0URCxYsQEhICDw9PREbG4vjx48b1KHVajF79mwEBATAy8sL99xzDy5evGjrj2IVKSkpGD9+PEJCQiAIAtauXWtwXKr7c+3aNTz00ENQqVRQqVR46KGHUFRUZOVPJ63b3av4+Ph637X+/fsbnOMM9yoxMRF9+vSBj48PAgMDMXHiRGRkZBicw+/VDcbcL363ai1atAg9evTQT1w2YMAA/Prrr/rjdv+9Ep3UypUrRTc3N/Grr74S09PTxaeeekr08vISL1y4IHdoNvPKK6+I3bp1E3Nzc/Wv/Px8/fG33npL9PHxEVetWiUePXpUnDJlihgcHCxqNBr9OTNmzBBbt24tbtmyRTx06JA4bNgwMTo6WqyurpbjI0lq48aN4rx588RVq1aJAMQ1a9YYHJfq/owePVqMiooSd+/eLe7evVuMiooSx40bZ6uPKYnb3atp06aJo0ePNviuFRYWGpzjDPdq1KhR4tKlS8Vjx46JaWlp4tixY8U2bdqIJSUl+nP4vbrBmPvF71at9evXixs2bBAzMjLEjIwM8cUXXxTd3NzEY8eOiaJo/98rp01G+vbtK86YMcNgX+fOncUXXnhBpohs75VXXhGjo6MbPKbT6cSgoCDxrbfe0u+rqKgQVSqV+Pnnn4uiKIpFRUWim5ubuHLlSv05OTk5okKhEDdt2mTV2G3t5h9Yqe5Penq6CEDcu3ev/pw9e/aIAMSTJ09a+VNZR2PJyIQJExot46z3Kj8/XwQgJicni6LI79Xt3Hy/RJHfrVtp3ry5+PXXXzvE98opH9NUVlbi4MGDGDlypMH+kSNHYvfu3TJFJY/Tp08jJCQEERERuO+++3Du3DkAQGZmJvLy8gzukVKpxNChQ/X36ODBg6iqqjI4JyQkBFFRUU3+Pkp1f/bs2QOVSoV+/frpz+nfvz9UKlWTu4c7duxAYGAgOnXqhMceewz5+fn6Y856r9RqNQDA398fAL9Xt3Pz/bqO3y1DNTU1WLlyJUpLSzFgwACH+F45ZTJy5coV1NTUoFWrVgb7W7Vqhby8PJmisr1+/fph2bJl+O233/DVV18hLy8PAwcORGFhof4+3Ooe5eXlwd3dHc2bN2/0nKZKqvuTl5eHwMDAevUHBgY2qXsYFxeH77//Htu2bcP777+PAwcOYPjw4dBqtQCc816Jooi5c+di8ODBiIqKAsDv1a00dL8AfrfqOnr0KLy9vaFUKjFjxgysWbMGXbt2dYjvlUOs2mstgiAYbIuiWG9fUxYXF6d/3717dwwYMADt27fHN998o+8AZs49cqb7KMX9aej8pnYPp0yZon8fFRWF3r17o23bttiwYQMmTZrUaLmmfK9mzZqFI0eOYNeuXfWO8XtVX2P3i9+tGyIjI5GWloaioiKsWrUK06ZNQ3Jysv64PX+vnLJlJCAgAC4uLvUyufz8/HqZozPx8vJC9+7dcfr0af2omlvdo6CgIFRWVuLatWuNntNUSXV/goKCcPny5Xr1FxQUNOl7GBwcjLZt2+L06dMAnO9ezZ49G+vXr8f27dsRGhqq38/vVcMau18Ncebvlru7Ozp06IDevXsjMTER0dHRWLhwoUN8r5wyGXF3d0evXr2wZcsWg/1btmzBwIEDZYpKflqtFidOnEBwcDAiIiIQFBRkcI8qKyuRnJysv0e9evWCm5ubwTm5ubk4duxYk7+PUt2fAQMGQK1WY//+/fpz9u3bB7Va3aTvYWFhIbKzsxEcHAzAee6VKIqYNWsWVq9ejW3btiEiIsLgOL9Xhm53vxrirN+thoiiCK1W6xjfK4u6vzqw60N7Fy9eLKanp4tPP/206OXlJZ4/f17u0Gzm2WefFXfs2CGeO3dO3Lt3rzhu3DjRx8dHfw/eeustUaVSiatXrxaPHj0q3n///Q0OBQsNDRW3bt0qHjp0SBw+fHiTGdpbXFwsHj58WDx8+LAIQPzggw/Ew4cP64d/S3V/Ro8eLfbo0UPcs2ePuGfPHrF79+4ONaRQFG99r4qLi8Vnn31W3L17t5iZmSlu375dHDBggNi6dWunu1dPPPGEqFKpxB07dhgMRS0rK9Ofw+/VDbe7X/xu3ZCQkCCmpKSImZmZ4pEjR8QXX3xRVCgU4ubNm0VRtP/vldMmI6Ioip9++qnYtm1b0d3dXezZs6fBcDFncH2cuZubmxgSEiJOmjRJPH78uP64TqcTX3nlFTEoKEhUKpXikCFDxKNHjxrUUV5eLs6aNUv09/cXPT09xXHjxolZWVm2/ihWsX37dhFAvde0adNEUZTu/hQWFooPPvig6OPjI/r4+IgPPvigeO3aNRt9Smnc6l6VlZWJI0eOFFu2bCm6ubmJbdq0EadNm1bvPjjDvWroHgEQly5dqj+H36sbbne/+N264eGHH9b/nrVs2VK866679ImIKNr/90oQRVG0rG2FiIiIyHxO2WeEiIiI7AeTESIiIpIVkxEiIiKSFZMRIiIikhWTESIiIpIVkxEiIiKSFZMRIiIikhWTESIiIpIVkxEiIiKSFZMRIiIikhWTESIiIpIVkxEiIiKS1f8D0W7XZgHjGP8AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses_sgd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1df5b665",
   "metadata": {},
   "source": [
    "接下来使用 Adagrad 优化器（在[官方文档](https://pytorch.org/docs/stable/optim.html)中找到对应的函数），其他参数保持不变，重新训练一次模型，也保存下来损失函数值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e8381b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0, batch 0, loss = 2.309314012527466\n",
      "epoch 0, batch 20, loss = 1.132602334022522\n",
      "epoch 0, batch 40, loss = 0.7322469353675842\n",
      "epoch 0, batch 60, loss = 0.6276703476905823\n",
      "epoch 0, batch 80, loss = 0.47763657569885254\n",
      "epoch 0, batch 100, loss = 0.37131136655807495\n",
      "epoch 0, batch 120, loss = 0.35471802949905396\n",
      "epoch 0, batch 140, loss = 0.2749488055706024\n",
      "epoch 0, batch 160, loss = 0.32309094071388245\n",
      "epoch 0, batch 180, loss = 0.29667168855667114\n",
      "epoch 0, batch 200, loss = 0.26814109086990356\n",
      "epoch 0, batch 220, loss = 0.32649776339530945\n",
      "epoch 0, batch 240, loss = 0.3371853530406952\n",
      "epoch 0, batch 260, loss = 0.2535422146320343\n",
      "epoch 0, batch 280, loss = 0.23318257927894592\n",
      "epoch 1, batch 0, loss = 0.21096524596214294\n",
      "epoch 1, batch 20, loss = 0.16355761885643005\n",
      "epoch 1, batch 40, loss = 0.2624760866165161\n",
      "epoch 1, batch 60, loss = 0.4048718214035034\n",
      "epoch 1, batch 80, loss = 0.2754383981227875\n",
      "epoch 1, batch 100, loss = 0.18976618349552155\n",
      "epoch 1, batch 120, loss = 0.26859599351882935\n",
      "epoch 1, batch 140, loss = 0.19074195623397827\n",
      "epoch 1, batch 160, loss = 0.2095036655664444\n",
      "epoch 1, batch 180, loss = 0.22769063711166382\n",
      "epoch 1, batch 200, loss = 0.22470030188560486\n",
      "epoch 1, batch 220, loss = 0.20652779936790466\n",
      "epoch 1, batch 240, loss = 0.20332807302474976\n",
      "epoch 1, batch 260, loss = 0.16568249464035034\n",
      "epoch 1, batch 280, loss = 0.13383789360523224\n"
     ]
    }
   ],
   "source": [
    "nepoch = 3\n",
    "batch_size = 200\n",
    "lr = 0.001\n",
    "\n",
    "np.random.seed(123)\n",
    "torch.manual_seed(123)\n",
    "\n",
    "model = MyModel()\n",
    "losses_Adagrad = []\n",
    "\n",
    "# 完成模型训练\n",
    "opt = torch.optim.Adagrad(model.parameters(), lr=lr)\n",
    "\n",
    "n = x.shape[0]\n",
    "obs_id = np.arange(n)  # [0, 1, ..., n-1]\n",
    "# Run the whole data set `nepoch` times\n",
    "for i in range(nepoch):\n",
    "    # Shuffle observation IDs\n",
    "    np.random.shuffle(obs_id)\n",
    "\n",
    "    # Update on mini-batches\n",
    "    for j in range(0, n, batch_size):\n",
    "        # Create mini-batch\n",
    "        x_mini_batch = x[obs_id[j:(j + batch_size)]]\n",
    "        y_mini_batch = y[obs_id[j:(j + batch_size)]]\n",
    "        # Compute loss\n",
    "        pred = model(x_mini_batch)\n",
    "        lossfn = torch.nn.NLLLoss()\n",
    "        loss = lossfn(torch.log(pred), y_mini_batch)\n",
    "        # Compute gradient and update parameters\n",
    "        opt.zero_grad()\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "        losses_Adagrad.append(loss.item())\n",
    "\n",
    "        if (j // batch_size) % 20 == 0:\n",
    "            print(f\"epoch {i}, batch {j // batch_size}, loss = {loss.item()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c00718ca",
   "metadata": {},
   "source": [
    "对比 SGD 和 Adagrad，画出各自的损失函数曲线。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3eb0e375",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(losses_sgd)\n",
    "plt.plot(losses_adagrad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b382b90",
   "metadata": {},
   "source": [
    "最后再自行选择一款优化器，重复上面的实验，并画出三种优化器的损失函数值对比图。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffc244f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "nepoch = 3\n",
    "batch_size = 200\n",
    "lr = 0.001\n",
    "\n",
    "np.random.seed(123)\n",
    "torch.manual_seed(123)\n",
    "\n",
    "model = MyModel()\n",
    "\n",
    "\n",
    "# 完成模型训练和作图\n",
    "losses_Adam = []\n",
    "\n",
    "# 完成模型训练\n",
    "opt = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "\n",
    "n = x.shape[0]\n",
    "obs_id = np.arange(n)  # [0, 1, ..., n-1]\n",
    "# Run the whole data set `nepoch` times\n",
    "for i in range(nepoch):\n",
    "    # Shuffle observation IDs\n",
    "    np.random.shuffle(obs_id)\n",
    "\n",
    "    # Update on mini-batches\n",
    "    for j in range(0, n, batch_size):\n",
    "        # Create mini-batch\n",
    "        x_mini_batch = x[obs_id[j:(j + batch_size)]]\n",
    "        y_mini_batch = y[obs_id[j:(j + batch_size)]]\n",
    "        # Compute loss\n",
    "        pred = model(x_mini_batch)\n",
    "        lossfn = torch.nn.NLLLoss()\n",
    "        loss = lossfn(torch.log(pred), y_mini_batch)\n",
    "        # Compute gradient and update parameters\n",
    "        opt.zero_grad()\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "        losses_Adam.append(loss.item())\n",
    "\n",
    "        if (j // batch_size) % 20 == 0:\n",
    "            print(f\"epoch {i}, batch {j // batch_size}, loss = {loss.item()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "411e0bfe",
   "metadata": {},
   "source": [
    "### 2.6 实施预测"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cc09b3f",
   "metadata": {},
   "source": [
    "为了验证模型的效果，我们对10个测试观测（即之前生成的 `testx`）进行预测。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "80a99a6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.    0.    0.    0.    0.994 0.    0.    0.    0.    0.006]\n",
      " [0.    0.992 0.001 0.    0.001 0.    0.002 0.001 0.002 0.   ]\n",
      " [0.    0.    0.998 0.001 0.    0.    0.    0.    0.    0.   ]\n",
      " [0.    0.    0.    0.    0.    0.    0.    1.    0.    0.   ]\n",
      " [0.    0.    0.    0.    0.    0.    0.999 0.    0.    0.   ]\n",
      " [0.999 0.    0.    0.    0.    0.    0.    0.    0.    0.   ]\n",
      " [0.    0.    0.211 0.785 0.    0.001 0.    0.001 0.001 0.   ]\n",
      " [0.    0.    0.    0.    0.    0.    0.    0.963 0.    0.037]\n",
      " [0.    0.    0.    0.    1.    0.    0.    0.    0.    0.   ]\n",
      " [0.    0.    0.    0.    0.    0.    1.    0.    0.    0.   ]]\n",
      "tensor([4, 1, 2, 7, 6, 0, 3, 7, 4, 6])\n"
     ]
    }
   ],
   "source": [
    "ypred = model(xtest)\n",
    "print(np.round(ypred.detach().cpu().numpy(), 3))\n",
    "print(ytest)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38c93ff3",
   "metadata": {},
   "source": [
    "如果模型搭建和训练都正常，那么每一行中概率最大的取值所在的位置应该正好对应真实的标签。我们也可以让 PyTorch 自动找到最大值的位置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e7bd7b9b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([4, 1, 2, 7, 6, 0, 3, 7, 4, 6])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.argmax(ypred, dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c34de42",
   "metadata": {},
   "source": [
    "最后，我们用模型对一些真实的手写数字图片进行预测。请你利用绘图软件（如 Windows 自带的绘图，或 Photoshop 等）准备10张正方形黑色底色的图片，每张用鼠标绘制一个数字（**请使用较粗的笔划**），从0到9，然后以0.png，1.png等文件名存储下来，放到当前目录一个名为 digits 的文件夹中。以下是几个例子：\n",
    "![](digits/sample0.png) ![](digits/sample5.png) ![](digits/sample8.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35066889",
   "metadata": {},
   "source": [
    "接下来利用 Pillow 软件包读取图片："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0d19fbd2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAADIAAAAyCAYAAAAeP4ixAAABPUlEQVR4nO2ZwRLCMAhEwfH/f7kenM501LTALoSx2YuHVsILJIVERWSTP9BjtgMsLZBuelYPuG3fS1JVYbsq5MX+y1GRt7OjZ8d3oqKBXDlpVRQGBmEBHBWBgRZ7BkTUbigiWQCf8kTGHREEwpsynrHM2y8ahR1i/2VH1ZRaoZw1zr7FtsXWZWp5IVTVlUKMj6EIsUTxAnz+F9UpiDUarFlFBEUEiYJHlgkdgmTWRRlqU8ajE9MGBFUIpFtaiQxAqmoppu6bWh3TSuTOEemq0n4kU20igk5QGxBUw8aqqtYqa6wQByps7DptdbP6EQ+A1Tbl7Pfo2Gjg7N3u8vBh5nZLPdeaVZJ4xzUt9mqYyHiuI9P0PK++VqCfEs686GHAMFO2/KIna73Rr95m6W+KxgXSTQukmxZINy2QbnoB9Fp0W2BMQcIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=50x50>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from PIL import Image\n",
    "im = Image.open(\"digits/sample0.png\")\n",
    "im"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f30128c",
   "metadata": {},
   "source": [
    "此时如果直接将其转为 Numpy 数组会得到三个通道："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f3999832",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50, 50, 4)\n"
     ]
    }
   ],
   "source": [
    "im_arr = np.array(im)\n",
    "print(im_arr.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06d85450",
   "metadata": {},
   "source": [
    "因此，我们先强制转换为灰度图片（单通道），再缩放至模型的图片大小 28 x 28："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "102618e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 28)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABO0lEQVR4nM2SzyvDcRjH35tZY98lBwdlK0rWDi5y4Sby6x9wQslVuVBOUo6LE2ft4Ook5GJJ4iJbSsiSLawo1pSxl8P2/e67jYuT9+nd83r6PO/P8/lIf5XD5jt8SvbOaDuSrGoLrL3CdQZIhCqQayKLpZtgGWs+/CzUTwC4arOxuTsAzncXfMNRgHCNxcYBuJxqkCSt5uC5zmTeMyC76THHP0K6yYSTwFOPdU59CvKLkuSUNCIpfWTB97DkcBeh0ShppZSOF0kUYahfysZt2Z02k5d0cFy1sVJXa+AX6JQUbLfVZiW5it6IAns+E3VGvyBjbXcU4HTaL8ntX74HWJdUeE/P1qAk5SIP6uqrlaTUwIU1xNihTKlueyzvxlsJ3S75K1IPxeIAfMTmW6yi7Q+NGZIS+z9d+L/oGzu/pEbyUulUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "im = im.convert(\"L\")\n",
    "im.thumbnail((28, 28))\n",
    "im_arr = np.array(im)\n",
    "print(im_arr.shape)\n",
    "im"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3978a9d6",
   "metadata": {},
   "source": [
    "为了传递给模型对象，还需要先将数值归一化到 [0,1] 区间，转换为 PyTorch 的 Tensor 类型，并增加一个批次和一个通道的维度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ce6af013",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 1, 28, 28])\n"
     ]
    }
   ],
   "source": [
    "test0 = torch.tensor(im_arr / 255.0, dtype=torch.float32).view(1, 1, 28, 28)\n",
    "print(test0.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69402359",
   "metadata": {},
   "source": [
    "最后对图片标签进行预测："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "28b1f088",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.288 0.    0.71  0.001 0.    0.    0.001 0.    0.    0.   ]]\n"
     ]
    }
   ],
   "source": [
    "pred0 = model(test0)\n",
    "print(np.round(pred0.detach().cpu().numpy(), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ebafef2",
   "metadata": {},
   "source": [
    "预测结果是否符合真实情形？请对你自己绘制出的10张图片进行类似的预测操作，并评价其效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "9fadb617",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image: sample0.png, Predicted Probabilities: [[0.11727883 0.08795216 0.17891951 0.08800594 0.08795216 0.08799057\n",
      "  0.08803775 0.0879527  0.08795804 0.0879524 ]], Predicted Label: 2\n",
      "Image: sample5.png, Predicted Probabilities: [[0.08551217 0.08551216 0.08562058 0.08652245 0.08551224 0.2291711\n",
      "  0.08552422 0.08551216 0.0856005  0.08551235]], Predicted Label: 5\n",
      "Image: sample8.png, Predicted Probabilities: [[0.08867452 0.08867091 0.08892094 0.14032996 0.08868616 0.09137246\n",
      "  0.0886739  0.08867089 0.14730373 0.08869645]], Predicted Label: 8\n"
     ]
    }
   ],
   "source": [
    "from PIL import Image\n",
    "import os\n",
    "\n",
    "png_folder = \"digits\"\n",
    "predictions = []\n",
    "\n",
    "for filename in os.listdir(png_folder):\n",
    "    # 读取图片\n",
    "    image_path = os.path.join(png_folder, filename)\n",
    "    im = Image.open(image_path)\n",
    "    \n",
    "    # 转换为灰度图并缩放\n",
    "    im = im.convert(\"L\")\n",
    "    im.thumbnail((28, 28))\n",
    "    \n",
    "    # 转换为模型输入格式\n",
    "    im_arr = np.array(im)\n",
    "    test_image = torch.tensor(im_arr / 255.0, dtype=torch.float32).view(1, 1, 28, 28)\n",
    "    \n",
    "    # 模型预测\n",
    "    prediction = model(test_image)\n",
    "    \n",
    "    # 获取预测概率分布和标签\n",
    "    pred_probabilities = torch.nn.functional.softmax(prediction, dim=1).detach().cpu().numpy()\n",
    "    pred_label = torch.argmax(prediction, dim=1).item()\n",
    "    \n",
    "    # 记录预测结果\n",
    "    predictions.append((filename, pred_probabilities, pred_label))\n",
    "\n",
    "# 打印详细的预测结果\n",
    "for filename, pred_probabilities, pred_label in predictions:\n",
    "    print(f\"Image: {filename}, Predicted Probabilities: {pred_probabilities}, Predicted Label: {pred_label}\")\n",
    "# 请在此处完成代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea5af41c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
